1 bug, $50,000+ in bounties, how Zendesk intentionally left a backdoor in hundreds of Fortune 500 companies
Posted by donutloop@reddit | programming | View on Reddit | 161 comments
m00nh34d@reddit
As per https://support.zendesk.com/hc/en-us/articles/8187090244506-Email-user-verification-bug-bounty-report-retrospective
That isn't what happened at all. Researcher disclosed it, and it was responded to as informational. That's done then, ZenDesk have decided it's not important now.
The next thing the researcher did was discover a full supply chain attack, which utilises this bug. The original bug hadn't changed at this stage, there was nothing new to disclose. What had changed was specific companies' implementation of different software suites leaving them vulnerable. It would be immoral at this stage not to disclose that vulnerability to those companies.
Maybe ZenDesk are right in their mind that this bug wasn't a big deal (they're not). But there was also an exploit with Slack OAuth here as well, which also got off scot-free. Both of those on their own might be insignificant to vendors, but when combined and presented to a company using both platforms, it does become a big risk for them.
Buo-renLin@reddit
Technically the researcher can still attempt to escalate this issue via in-band methods(HackerOne), before actually disclose to the search party.
sanylos@reddit
I find it ridiculous.They drop it as out of scope, you try to alert their customers, they finally realize the danger of the exploit and give you a 0$ because you raised awareness of the threat with their customers
ozyx7@reddit
The original issue he reported was out of scope per the terms of the bug bounty. While those terms are unfortunate, the simple fact remains that it was out of scope.
He then leveraged his issue with another issue (Zendesk not blocking automated email from
appleid@id.apple.com
) that he did not report to Zendesk. (Arguably it shouldn't be Zendesk's responsibility to block every possible sender email address used by every possible OAuth provider used by arbitrary third-party services such as Slack, but he didn't try to demonstrate this to Zendesk to prove how serious the original issue was.)rabid_briefcase@reddit
This is something many executives need to learn.
Pay the white hats generously,or you will pay far more when the black hats do the same work. The only way to be notified is to pay for all of them, not to announce an exclusion to not pay out.
It is now likely with a PR backlash that they will pay out, but it is not before the company has paid with a PR problem and multiple lost contracts described in the article. They saved the money on a bounty in the short term, but are paying far more as a result, and will likely pay again as gray hats won't hesitate to get money from the bad actors after exploits like this are announced.
Doctor_McKay@reddit
It's not the security researcher's job to come up with every possible way a vulnerability can be exploited. The researcher demonstrated information disclosure, and that's enough to prove a vuln exists.
ozyx7@reddit
I'm talking specifically about eligibility for the bounty. I'm not claiming that information disclosure isn't a serious problem.
The second issue maybe would have been eligible had he reported it.
Doctor_McKay@reddit
That's the thing though, it's the same issue.
ozyx7@reddit
It's not the same issue. The first issue is what allows the second issue to exist. If Zendesk blocked
appleid@id.apple.com
like it blockednoreply@...
senders, then he wouldn't have been able to do what he did.Again, I'm not downplaying the first issue. Unintended information disclosure should have been treated seriously. I also don't personally agree with the terms of the bug bounty, but those terms are what they are.
Doctor_McKay@reddit
The reported information disclosure vuln is the one and only issue. Without ID, it doesn't much matter if they block appleid@id.apple.com or not.
N3RO-@reddit
Next time, sell it to malicious actors. xD
Fuck companies who do these bad practices in bug bounty!
Bekwnn@reddit
And screw over every company that happens to use Zendesk and possibly to those companies' customers? Do random harm to ostensibly innocent and uninvolved parties?
Why the hell is this upvoted?
N3RO-@reddit
This is not Red Cross or Our Lady of Mercy. Bug Bounty is a business, just like Zendesk. They played dirty with the researcher, so fuck them.
I managed many H1 bug bounties in the past, and we never did shit like that. Pay your researchers accordingly when the finding is legit.
Bekwnn@reddit
And selling this doesn't just fuck over Zendesk, it fucks over a lot of other random people who don't have any say in how Zendesk handles their bug bounties.
Don't be a shitty person and sell exploits that would inflict harm to a bunch of different parties because one party slighted you.
1esproc@reddit
This is the nature of SaaS, you as a customer are making a decision to absolve yourself of risk, and in doing so you as a company should make a prospective vendor's bug bounty track record as part of your risk profile. Do not buy software from companies that do this because you are putting your data and operations at risk.
Bekwnn@reddit
In that regard, I don't find what the user who posted the article did wrong.
People suggesting to sell the exploits to criminals, so that the backdoor can be exploited and those companies can randomly get pantsed for using what they thought was a widely used and trusted 3rd party software, are being crappy.
It's not about being saint. It's about not empowering criminals to randomly commit identity theft, ip theft, and fraud against other random companies whose only apparent crime is using Zendesk.
It's not about being a saint. It's about not selling lock picks to criminals because a company that sells locks slighted you.
IsABot@reddit
Then by that logic, Zendesk should have just paid out the bug bounty to the guy when first reported and they verified it gave access. They should have done right by their own customers who they owe it to. Instead they used the excuse of "email spoofing" doesn't count. You think hackers/criminals don't spoof emails/ips/phone numbers all the time? It should always be in scope if someone can easily fake basic information to get into a system they shouldn't have access to. They reported an issue to them first, responsibly, so they should have been paid, even if only partially due to "not meeting every requirement exactly". What they have shown is that honesty is not rewarded.
Bekwnn@reddit
By what logic?
My whole point is that you shouldn't be causing harm to random Zendesk customers because of what Zendesk did.
Yes it would cause harm to Zendesk as well but that doesn't in any way excuse the collateral. Driving a truck into a crowd to hit one person.
I'm not excusing Zendesk at all here, I'm saying that the original comment I replied to is taking a shitty stance, selling exploits to criminals that would catch random people up in harms way.
And the fact that people can't see that here, and are even upvoting it and downvoting the sane responses calling it out is insane.
But this is reddit and no one reads the damn article so let me lay it out for everyone who didn't:
Zendesk had an exploit, the exploit was proven to compromise the Slack and ticket systems of every company that uses Zendesk.
Zendesk said, "thank you for bringing it to our attention, but not a bug bounty," ostensibly downgrading the severity of the exploit, and quite possibly backlogging it, leaving Zendesk's customers exposed and at risk.
The bug hunter/article author reached out to those companies exposed with the exploit, notifying them and recieving individual bug bounties from those Zendesk customers. Some of those Zendesk customers stopped using Zendesk.
The article author helped the people Zendesk was putting at risk, and it seemingly caused some blowback as some of them likely stopped being Zendesk customers.
Everybody lived happily ever after (except Zendesk).
The above comment I originally replied to wanted to instead screw over other companies using Zendesk for the crime of using Zendesk.
They suggested selling the exploit to criminals to steal and defraud other companies who aren't Zendesk.
All for the primary purpose of getting paid by criminals, because Zendesk didn't pay them.
Hopefully that makes the moral failing more clear.
IsABot@reddit
The logic of, if you don't want people to turn to criminal methods then take care of them when they were looking out for you and your company/clients first. Instead they swept it under the rug, and thus OP (of the article) went to escalate it to people that would care about it. You don't get to fuck over someone then complain when they fuck you back.
Bekwnn@reddit
You're not reading any of my comments properly and replying to things I never said.
If they sold them to criminals they would be catching up other companies and people in harms way, not just Zendesk.
It would be a shitty thing to do.
Why are you still acting like I'm defending Zendesk in any way?
I don't have any issue with what the OP article writer did.
I have an issue with the crappy people in this comment section suggesting to sell the exploit to criminals, which would then put uninvolved parties (again, NOT Zendesk) at risk of fraid, ip theft, identity theft, etc.
Consider rereading my comment and/or the article because you seem to be missing it.
IsABot@reddit
No, I read your comments. You just aren't getting it. You are talking about not selling to criminals. It's not the client's fault for zendesk's action. Yeah, I agree with that. But if your logic is saying "hey don't do shitty things to innocent people", then by that very same line, you should hold zendesk to that very same standard, which you aren't.
Because you are by proxy defending what zendesk did and are simply trying to handwave that part away and just say "don't do shitty things to the poor corporation". It's all connected. Zendesk shouldn't have done what they did to the innocent person who tried to help them. Instead they threw them under the bus, which also throws every one of their clients under the bus at the exact same time by trying to bury an actual issue and get out of paying.
You don't get to try to claim someone else has to take the high road when the corporation takes the low road. So if you don't want people to sell to criminals, then pay them for their work before it gets out. If the OP hadn't found it, then it was only a matter of time before some bad party did. So again the company should be happy to pay that OP. But they weren't. And you aren't fighting for OP, you are just saying, oh well it doesn't matter they got screwed, just don't hurt anyone else.
The person saying fuck the corporation, sell to the criminals, is simply pointing out that if they don't care to pay, then why should they care? The thing you aren't putting together is the rest of society is quickly becoming tired of the corporate bullshit where they can fuck everyone else over for a buck, but then you expect people to just sit like good animals and take it. The only way these fuckface corporations learn anything is by holding their asses to the fire and sometimes collateral damage is unavoidable. At multiple points, Zendesk fucked up and in turn screwed their own clients and made them collateral damage in the first place. Yet you aren't talking about that at all, just "oh won't someone please think of the innocent clients".
Bekwnn@reddit
I'm not. That's in your head. And a big stretch of anything I said.
I'm defending the Zendesk's customers who are exposed by Zendesk's mistakes, and who you want to screw again to get a pay day.
You're literally trying to fuck over random uninvolved parties for a quick buck.
Why stop there? Go rob a store. Defraud a retirement fund. You're tired, so fuck everyone else right? The sky's the limit.
But hey if you can make a few thousand by helping twist the knife in their gut, it's your due right?
You don't know anything about the parties you'd be harming besides the fact that they're companies. But sounds like that's enough feel justified.
Bekwnn@reddit
To one more time try dumbing it down further, you have:
Party A: Bug Reporter (Article author)
Party B: Zendesk
Party C: Every company that happens to use Zendesk
Selling to criminals would inflict harm on Party C, who is completely unaware and uninvolved in the entire situation.
Party C has no knowledge, say, or idea of how Party B (Zendesk) is handling their bug bounty program.
You'd be driving a truck into a crowd full of strangers (Party C) to knock over Zendesk's lemonade stand (Party B).
And everyone who's saying, "Good, drive the truck into the crowd of strangers, that'll show Zendesk," is a shitty person.
OP instead did the right thing of reaching out to notify Zendesk customers (Party C) when Zendesk (Party B) refused to take action.
Maybe /r/programming just has a lot of sociopaths.
IsABot@reddit
Well aware of the situation. Maybe you can tell better though by standing on that soap box.
Deranged40@reddit
Make no mistake, after Zendesk has pulled this stunt, me choosing to sell my new knowledge is not me fucking over Zendesk's customers, It's Zendesk fucking over those random people a second time.
This act that we're talking about, where they pull the bug and then offer $0 after their customers bitched about it is Zendesk fucking over every single one of those random people that have a huge say in who they choose to use for SaaS provider.
Tired8281@reddit
Because they are right. Zendesk had many opportunities to do the right thing, and they repeatedly chose to sacrifice their customers. If this dude found it, so could have the bad guys.
Uristqwerty@reddit
If you understand society at a systems level, you'd see that becoming a black hat hacker in retaliation undermines any influence you could have had. Stopping an individual hacker then counts as a solved problem in its own right, with no need to address whatever underlying issue motivated them.
Want to influence companies to improve? Don't attack customers/users, directly or indirectly. That gives them an excuse to take on the role of defender, protecting customers from the evil hacker. Documenting how hard you tried to work with them to protect users, and showing how the company was the one who failed is necessary to make sure all the reputation damage lands on them properly, maximizing the motivation to improve.
unkz@reddit
Where in this scheme do I get paid?
Uristqwerty@reddit
Hopefully from future bounties, from companies who learn from the PR disaster.
It won't be a PR disaster if you go the black hat route and sell the vulnerability to third parties, though, so other companies won't learn. "It's a hacker. Nothing we could do; hackers gonna hack."
Plus there's the risk of negative money if caught.
PaintItPurple@reddit
This would require a poor bug bounty program to constitute a "PR disaster." Given that this has never happened in the entire history of the world, that seems pretty fanciful.
Uristqwerty@reddit
The "PR disaster" is what follows a writeup of how shitty the company acted, posted as a blog.
Like the very reddit post that was submitted here. And numerous others over the past decades, that each seriously harmed a company's reputation in the eyes of the security research community.
PaintItPurple@reddit
A post on r/programming with a 100 comments is not a PR disaster. In three days, nobody is even going to remember this post existed. You'd literally be better off hoping the Ghost of Christmas Past changed their minds.
Uristqwerty@reddit
Correct. At least not on its own. The post on reddit is just one of who-knows-how-many social media links, a seed for virality as the story spreads gradually. It's the spread as a whole, which becomes a disaster that'll influence the company's behaviour.
And even if only a hundred redditors ever see it? Still more impact than selling the exploit on the darkweb would get, where chances are the company silently fixes the issue and the public never even hears of it.
PaintItPurple@reddit
None of what you're fantasizing has ever happened and there's no reason to believe it will magically start happening now. Don't believe me? Watch as this story's "viral spread" looks much more like a "fizzle out," just like every article about a shitty bug bounty program before it.
Uristqwerty@reddit
Fantasizing? I'm referring to a number of articles that got posted to this very subreddit over the course of the past decade, and how negative reputations about their bounty programs lingered for what felt like a year or two following each, among the programming and security communities I browsed often at the times.
There's no fantasy about historic events. Unfortunately, though, I can't find the posts themselves. There are things like this /r/netsec submission with over a thousand upvotes, claiming to be a crosspost from /r/programming, but searching here for "bounty" turns up no evidence. Nor "paypal". So either the submission was removed for being non-programming, its submitter bulk-purged their posts in protest, or reddit search is being crap. But back in the subreddit's peak days, when most posts got hundreds of comments and thousands of votes? There were multiple news items per year about bug bounty shittiness, from what I personally recall seeing, and each left a lasting impact.
To say nothing of twitter back then.
unkz@reddit
Isn’t this kind of disproving your point? Yeah, shitty bug bounty programs have been exposed before, and they are still shitty. Those “PR disasters” didn’t do anything meaningful.
Uristqwerty@reddit
Only disproves it to people who expect absolute change rather than gradual influence, though that aligns well with people's attitudes about the zendesk post. To me, it's successful pressure, made all the more successful the wider it spreads. To others, apparently, it's motivation to give up on bug bounties entirely, and instead actively harm customers whose only sin was trusting a widely-used company.
UpstageTravelBoy@reddit
"Rely on the other parties good behavior" is pretty bad game theory
Uristqwerty@reddit
It's not relying on good behaviour, it's recognizing that becoming a bad actor yourself will provoke worse behaviour in return, not pressure them to improve.
Before bug bounty programs became commonplace, the status quo was companies not caring much about security anyway, and using the legal system to punish hackers when they could catch them.
RICHUNCLEPENNYBAGS@reddit
It sounds like he still managed to collect $50,000 from affected companies, without committing any felonies
Empty-Win-5381@reddit
Yeah, jail exists
neumaticc@reddit
That's what these stupid policies encourage.
from what I've read on HN, a hackerone mediator deemed it not an issue, so ZD could have blamed them and not eaten shit publicly.
regardless, guy found an issue and they fixed it. that means there's some level of severity to it, and he should be fucking rewarded!
hellomistershifty@reddit
Yeah, if they outsourced the bounty program they need to own it themselves when the company they outsourced to fucks up
Empty-Win-5381@reddit
Yeah, so dumb, as though they'd be overwhelmed by the bounty program. I'm sure they paid more for the horrible outsourcing than they'd pay in bounties
destroyer1134@reddit
That's what has to happen. The only reason these bug bounty programs exist is so people can use their skills for good and make a living. if they take that away, people will find another way to get paid.
walen@reddit
No. The only reason these bug bounty programs exist is so companies can have their security bugs discovered and fixed by "good" actors before any "bad" actors manage to exploit them.
Companies couldn't care less if people are able to make a living out of it or not.
If bug bounties didn't exist, the only ones trying to find vulnerabilities in your code would be hackers trying to exploit them for potential gain (sell the exploit itself, sell your data, use it as a means to access other systems, etc.).
By putting out these programs, companies are just incentivizing "neutral" hackers to proactively find any holes in their systems, notify the company and then keep their mouths shut until it's fixed, in exchange for real $$$.
In other words, they are outsourcing their Security QA department. Way cheaper to pay 50k once a year to some random guy, than spending 300k/year on a couple of Security Engineers (or whatever the job title is). And, of course, it's way cheaper to pay 10k than to pay 50k, and way way cheaper to pay 0k.
The people claiming these bounties are not the same people who would sell the exploits on the darknet, and it's dumb to frame it like that. If they take the programs away, that people will just use other companies' programs, or get a more regular job if they don't already have one.
Empty-Win-5381@reddit
And also more efficient than paying a couple 300k security engineers, since you might have an inumerable amount of people working on it and who are very properly incentivized to actually provide something, find a bug
mallardtheduck@reddit
Which is a long way of saying it's in the best interests of the company to pay decent bounties. More "eyes" on their product, cheaper than employing a specialist team.
But lets not pretend there are no actors who might be tempted to sell an exploit to bad actors if there's no way to legitimately benefit from it (hell, in some cases "good" actors reporting an issue for no personal gain have been at least threatened with prosecution). The term "grey hat" wasn't invented by me...
double-you@reddit
It sounds like you are implying that people who work on bug bounties would turn to actual crime if the bounties didn't exist. And that bug bounties are just whitewashed blackmail.
draculamilktoast@reddit
Also so that companies can exploit you by simply not paying when you do the work. We are living in a world led by welchers who never get their comeuppances.
otac0n@reddit
When "what has to happen" is a crime, you are volunteering other people's bravery.
Slime0@reddit
Im unclear on this: didn't they just say it's out of scope for paying a bug bounty? That doesn't mean they didn't intend to fix it, right?
alexbirsan@reddit
They used the "informative" status, which translates to:
However, based on the screenshots in the article, his report was never actually seen by Zendesk staff, and the decision was taken by a middleman (HackerOne) on behalf of Zendesk.
source: been on both sides of this kind of interaction
xTeixeira@reddit
I disagree with this. Sure, they're technically justified within the scope of the third party platform's policy, but that ignores the fact that there is no justification for choosing a third party platform that has terrible policies, nor does it justify their own bad policy of SPF, DKIM and DMARC issues being ineligible for bounties.
Coffee_Ops@reddit
Are we ignoring that this issue has nothing to do with SPF, DKIM, or DMARC?
The issue is that zendesk itself blindly relies on the to Field to authenticate an attempt to add someone to a ticket, and that to field relies on a guessable, sequential ticket ID.
I have no doubt that the SPF, DKIM, and DMARC records for all the companies involved are impeccable. They had no bearing this issue.
afranke@reddit
FTA
Please do correct me if I'm wrong I don't consider myself an expert in these areas, but
1) If the "requestor’s email address" had SPF properly configured, you shouldn't be able to send a spoofed email from that domain unless you're on the correct IP.
2) If DKIM were configured and the spoofed email didn't have the correct key, this exploit would fail as the message would be rejected.
3) DMARC just needs to be configured correctly if messages pass the first two, so it seems it wasn't here.
Also FTA
Aren't those three technologies specifically meant to prevent e-mail spoofing?
Also, the screenshot in the article makes it clear it was rejected because it was a "SPF, DKIM, DMARC issue".
Empty-Win-5381@reddit
Would that mean this kid is a liar or something is left out?
afranke@reddit
No, just not detailed enough. We need to know how the emails are actually handled to determine where the issue is. If the companies are self-hosting e-mail (or using a service that isn't Zendesk but are in control of their own DNS for it), then its the fault of the company and they had an insecure configuration that was used against a Zendesk install (and could be abused in any other number of ways).
If Zendesk is hosting the e-mail service on behalf of the companies, then Zendesk screwed up DNS configs and essentially allowed this to happen.
Empty-Win-5381@reddit
Got it. How would the system prevent spoofing though? Guess I should learn more
afranke@reddit
SPF stands for Sender Policy Framework, and it's an email authentication method designed to prevent email spoofing. SPF allows the owner of a domain to specify which mail servers are permitted to send emails on behalf of that domain. If the "requestor" e-mail host has it properly configured in their DNS records (such as Google for Gmail, a requestor being a customer creating a support ticket) AND the receiver (Company/Zendesk in this case) properly VERIFY the SPF record it would be rejected if it came from the wrong server. This is mostly applicable to things like company e-mail addresses, beacuse if the requestor is on Gmail and you are also on Gmail, the spoofed mail is coming from the correct server (assuming Gmail lets you send messages as another gmail user, which I wouldn't think it does).
DKIM stands for DomainKeys Identified Mail, and it's an email authentication method designed to allow receiving mail servers to verify that an email has not been tampered with during transmission. It does this by attaching a digital signature to the email, which is linked to the sender’s domain, ensuring the integrity and authenticity of the message. Much like above, if the sender (like Gmail) has it properly configured its up to the receiver to verify it. Also like SPF above, if you're both sending from the same server, you both have the same DKIM key (again, assuming Gmail allows you to spoof at all before it's even sent).
DMARC stands for Domain-based Message Authentication, Reporting, and Conformance, and it’s an email authentication protocol that builds on SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail). DMARC allows domain owners to protect their domain from unauthorized use, such as email spoofing, by defining a policy that tells email receivers how to handle emails that fail SPF or DKIM checks. DMARC also provides reporting, so domain owners can monitor and track the use of their domain in email communications. And like the above, it's up to the receiver to verify this information, which should be contained in the DNS records of the sending domain. Here's Googles for example:
Apple:
So if my e-mail server receives a message from one of these domains, if I have it configured to, it will first check SPF to make sure the sender IP is an allowed IP, then DKIM to verify the key came from the correct domain, and if those fail it will follow the DMARC instructions to reject/quarantine and report. You can see this all happening in your e-mail headers, which are usually hidden from view but would look like this (big chunks cut out for relevance) form a message Apple sent to my Gmail:
Empty-Win-5381@reddit
This was remarkable. So sender must've been totally off since gmail is presumably very strongly configured not to allow spoofing and sending from their server like that, specially with the simple script he divulged. It would seem like info is off. The message apple sent to your email is pretty hard to decipher. I read reject and not permitted sender. You're awesome. You are the person that's made Reddit most worthy for me undoubtedly. These kind of thorough responses are completely beyond remarkable. Thank you for that. Do you work with cybersec?
afranke@reddit
Yup, I work on an Incident Response team, so digging through logs to figure out what happened when and where is one of my things.
In the headers I provided you can see:
X-Dkim_Sign_Required - not a standard flag AFAIK, but probably used for whatever mail server is receiving the message and indicates it requires a DKIM signature to accept an e-mail. This means any non-DKIM message is likely to immediately bounce from this server.
Authentication-Results line indicates an SPF "softfail" which means according to the SPF Apple has set up, even though this mail didn't come directly from an authorized IP, it can still go through. We also see "dkim=pass" on that line indicating we have passed the DKIM check.
Dkim-Signature line is just the DKIM signature Apple has configured.
-1st Received-Spf line - basically a more detailed version of the softfail telling us Google.com says "domain of transitioning no_reply@email.apple.com does not designate XXX.XXX.XXX.XXX as permitted sender" which means the IP isn't in the SPF record, but because of how the record is configured it's not an automatic fail
2nd Received-Spf line - a pass, which is likely why the other is a softfail. This one is telling us the mail server "mx-inbound19-53.us-east-2b.ess.aws.cudaops.com" (dunno who that belongs to) got the message from an authorized sender: domain of no_reply@email.apple.com designates 17.23.6.24 as permitted sender (as a random side note unrelated to this, Apple owns all 16,777,216 IP addresses starting with 17.x.x.x, so you can be sure any connection coming from that block is definitely Apple)
X-Bess-Dmarc - again not a standard flag, (I think this one is used by Barracuda Email Security Gateway), but it also indicates "no-action" and "SPF:Pass(s)" so that's a good thing.
I'll tell you, I learned most of this from typing up the comment. The lack of detail in the article raised some questions for me, so I had to find some answers as best I could. I was aware of these technologies beforehand, but never had a reason to really look into them other than to verify my company has them configured correctly (we do!) and that's usually done with a scanner of some type giving me a Yes or No answer.
I assume he was able to pull this off because the company receiving the support requests was likely hosting their own mail rather than paying someone like Google or Microsoft to host it, and because of that had a misconfiguration that allowed a spoofed message through.
So he sent an email to (for example) support+12345@companyA.com with a spoofed e-mail from requestor@anydomain.com with his own e-mail as a CC to get the system to add it onto the ticket. The key here is you have to be able to spoof the requestor e-mail address to add your own onto the ticket, and that can only be done if the receiving server DOESN'T verify those items. He spoofed an e-mail from "appleid@id.apple.com", any basic e-mail server in operation should be able to do a DNS lookup and verify SPF and DKIM records I've already verified Apple has in place.
tsimionescu@reddit
I think what's unclear here is what exactly is the email that Zendesk receives. If Zendesk is receiving the email from the attacker with the spoofed email address, then it's on Zendesk to validate. If the company's email server is the one that is receiving it and forwarding it to Zendesk, then it's on the company. Additionally, if the company doesn't have proper SPF/DKIM/DMARC for it's own emails, and thus they ask Zendesk to disable verifications for their domain, then it's again on the company.
So, it's possible that this is a problem with the company itself, but it can still be a problem with Zendesk. Not to mention, Zendesk could have thought ahead of time of a policy of rejecting emails at least from known large OAuth providers.
afranke@reddit
From what I can find, it's up to the customer to handle receiving messages:
https://support.zendesk.com/hc/en-us/articles/4408832543770-Allowing-Zendesk-to-send-email-on-behalf-of-your-email-domain
And to the credit of Zendesk, they do have an article detailing the needed mail server configs, including sections on SPF, DMARC, and DKIM: https://support.zendesk.com/hc/en-us/articles/4412991936922-Workflow-Ensuring-email-system-compatibility-with-Zendesk
nemec@reddit
Presumably if the company (not Zendesk) has set up DKIM correctly the spoofed email to add yourself to the ticket is denied by Zendesk.
Coffee_Ops@reddit
Presumably if that was the case, remediation for the customers he directly contacted would have been as easy as "fix your records", not "disable a zendesk feature."
nemec@reddit
If the company wasn't smart enough to set up SPF/DKIM/DMARC in the first place it's not a surprise they'd tend to overreact and place the blame anywhere but themselves.
PmMeUrTinyAsianTits@reddit
Seriously. Its gross how much "well they passed the buck" is accepted as an excuse for responsibility.
fuseboy@reddit
That's very interesting. I wasn't aware of that, is that an implication of their terms and conditions somehow?
alexbirsan@reddit
For HackerOne specifically, I think they cite their CoC, with the caveat that "vulnerability" can also refer to disputed, unvalidated, or otherwise invalid vulnerabilities.
This is standard across the board, for example Bugcrowd communicates this better by referring to them as "submissions", which encompasses any and all report contents.
Empty-Win-5381@reddit
Well, then they will sue the kid?
Empty-Win-5381@reddit
How did he waive that right? He signed no contract and I think he won't have legal troubles from this
knottheone@reddit
Did they take action on it? If so, it's not informative and they owe a bounty. That is the actual litmus test.
They did take action on it, they were working on a fix for two months and deployed it after two months. That highlights to me that it was actually being looked at even tagged as informative.
PartlyProfessional@reddit
He emailed zendesk and they just denied his appeal. And he wouldn’t receive a bounty from them, and decided to milk their contracted companies. Smart kid
Thirty_Seventh@reddit
The appeal was also rejected by HackerOne staff ("Patrick | H1 Mediation"). He didn't manage to get through to Zendesk until after contacting Zendesk clients.
to11mtm@reddit
So, basically Zendesk started listening when the finder realized the bureaucratic shitshow was on the level of Comcast and decided to do the morally correct thing?
H1+ZD are practicing security theater bolstered by kafkaesque bureaucracy.
SanityInAnarchy@reddit
This part seems incredibly shitty, to the point where it'd seem it defeats the purpose of a bug bounty program.
I mean, the purpose of a bounty program is, first, to incentivize security researchers to bring reports to you instead of selling it to malicious actors, and second, to actually make software more secure.
When that process is used to sit on a vulnerability instead of acting, obviously it makes everyone less secure, and it also removes any incentive for the researcher not to disclose anyway. After all, if it's going to be $0 bounty either way, there's no reason not to disclose it to third parties or publicly and still get $0. Or, for that matter, sell it to a malicious actor for more than $0.
jrochkind@reddit
Hm, marking it "informative" you still have waived yiour right to share the information with anyone else?
Based on how you describe "informative", that seems like a problem to be fixed. It seems to me marking an issue "informative" should mean this is not confidential/privileged information, it is information that is already publicly known or can be publicly known with no danger.
If they think otherwise, they should be giving you a bounty, right? That's what "informative" is for, no?
fordat1@reddit
Use-case clearly seems to be make the bug bounty program cost less money
JaggedMetalOs@reddit
I mean, playing devils advocate, Daniel did come up with a much more serious attack than his original report. But Zendesk really should have taken a defense in depth approach and just accepted and fixed the original bug report.
Empty-Win-5381@reddit
What do you mean, how did he come with a much more serious attack?
JaggedMetalOs@reddit
Leveraged the bug to gain access to private Slack channels via SSO.
Empty-Win-5381@reddit
How did his inclusion of himself on the CC give him access to the private slack channels?
JaggedMetalOs@reddit
The method is detailed in the article, basically he was able to use it to verify an AppleID account at support@company.domain and then access Slack as if he had a legit company email address.
Empty-Win-5381@reddit
I see. So he asked for it
tsimionescu@reddit
It seems there is more nuance than that. Per the disclosures, the original issue is still considered out of scope and nothing has been done about it: if it's possible to send spoofed emails from a company, then you can still get added to tickets. What they've done is to add more layers to prevent OAuth confirmation emails from automatically opening tickets like this. This is fixing a vulnerability that he never even tried to disclose to Zendesk.
Ultimately it's unclear to me what Zendesk's responsibility even is. If the company server is accepting and forwarding spoofed emails to Zendesk, why is it this Zendesk's problem? Why isn't this something the company itself should fix, not relying on Zendesk to add spam filters to protect them from their bad email practices?
mathisfakenews@reddit
this kid will go far
jherico@reddit
Maybe, maybe not.
Sometimes well meaning people who point out issues in vulnerable systems get hounded to death.
throwawaystedaccount@reddit
But this young man is a hacker, a tough cookie, and is more likely to be sought after as an ethical hacker due to his history of exploits on github and his ability to work with literally Fortune 500 at 15, and make $50,000 off the whole thing. Aaron Swartz was an idealist, equally effective yes, but he isn't pragmatic / practical in the way this young man is. I don't see a problem for this guy making it big, but I do see him being tested and tempted by money and power in the future. He already knows how to handle some types of power, so he seems to have a good start in that department.
knd775@reddit
You really don't know much about Aaron Swartz, then.
throwawaystedaccount@reddit
Yes, I think I need to read more about him. I knew only half the things he did.
Kirk_Kerman@reddit
He was looking down the barrel of a $1 million fine and 35 years in prison for the crime of downloading and redistributing JSTOR articles. Most research is in at least some part publicly funded, and paywalling it is morally heinous, never mind the very real costs of making future research harder and more expensive.
NoveltyAccountHater@reddit
Look, I agree that research papers should be free (especially those funded by gov't grants) and the Open Access movement. If I'm on a jury, he's an easy case to go for nullification. He was offered a plea deal of six months in minimum security prison for the downloads for pleading guilty (which his team rejected in favor of a trial).
That said like many other prodigies/geniuses, he did publicly write about his severe depression going back to 2007, well before the 2010-11 MIT JSTOR downloads and his 2011 arrest.
Empty-Win-5381@reddit
Yeah, High IQ and depression go hand in hand. Unless the person manages to mentally aspire to a higher World, they will be in big trouble. They absolutely CANNOT live among average or low IQ folks, lest they just see no point in the game of life and seek to obliterate themselves and others full of rage
RationalDialog@reddit
Exactly. don't be a hero and if, do it under the protection of a company that actually works in this area vs doing it as a hobby.
Prod_Is_For_Testing@reddit
He planted an illegal wiretap on campus to suck down data and distributed it for free. He got caught and couldn’t face the consequences. He’s not a saint. He was extremely misguided
GasolinePizza@reddit
What issue in vulnerable systems was Swartz exposing? It's not a vulnerability that he had access to research papers, not any more so than someone being able to film a movie is a vulnerability.
Aaron Swartz mass-pirating journals wasn't him announcing any vulnerabilities in a system, he was taking advantage of a benefit that was given in good faith.
You can argue whether his "information should be free" ideology is right or wrong and I won't take either side on that because that's tangential to my point: committing a crime in pursuit of what he believed in doesn't have anything to do with this article's case or the young man featured in it.
SpaceMonkeyAttack@reddit
What have they got against SPF and DKIM? Do they want spoofed emails?
nemec@reddit
Zendesk isn't responsible for
company.com
SPF and DKIMSpaceMonkeyAttack@reddit
But they also don't have to accept emails that didn't pass.
nemec@reddit
But they did pass. that's why the fix is to properly enable SPF/DKIM/DMARC, so that mail doesn't pass. There are "allow all" settings that would let spoofed mail pass or various other ways a company could ignorantly allow spoofed mail.
rdtsc@reddit
Everyone faulting ZenDesk for rejecting. It reads to me as if HackerOne was the one rejecting it, without any nuance. They even admit that it is a potential high-risk vulnerability and just sat on it? I would seriously rethink funneling all reports through HackerOne.
klo8@reddit
Yes, it's a HackerOne person replying to him, not ZenDesk. I imagine they have a policy against SPF/DKIM/DMARC issues because of beg bounties which lead to a lot of noise. Unfortunately it also caught a legitimate vulnerability here. This part I'd say is not on ZenDesk, that's just kinda unfortunate.
Nowaker@reddit
It wasn't on Zendesk until Zendesk learned about it, and decided to double down on HackerOne's view on this. https://support.zendesk.com/hc/en-us/articles/8187090244506-Email-user-verification-bug-bounty-report-retrospective
But in reality, if you outsource certain responsibilities to a third party, you cannot claim "it's on the third party". It's like claiming frivolous actions of Delta gate agent or flight attendants aren't on Delta. Sure they are. They act as an agent of the business, and are collectively responsible for the outcomes of their actions.
Yeah, and this lack of care for security resulted in multiple Zendesk customers leaving, as they maintain strict standards for responsible disclosure.
While beg bounties are clearly out of scope, this wasn't one. Thread forwarding is at the core of Zendesk's business. Zendesk & HackerOne fucked up.
ToughAd4902@reddit
I can't believe this is getting upvoted, it is so blatantly wrong.
They outsource the bug bounty program to HackerOne, and say we only want these specific things to be checked. There are a ton of reasons for this, from preventing people from spamming production servers, causing glitches, etc. You set a parameter list of what you want to be watched, and that's the only thing you want people to be looking at. This exploit should not have been tested from a white hack perspective, full stop, the author is in the wrong, regardless of how big or small of a vulnerability this was. This isn't even about sending out to the customers which was also VERY wrong as those customers can now literally use that to go and attack their competitors.
As he did find it, breaking the entire point of HackerOne, and he did want to report, he needed to escalate it to zendesk's internal team, NOT through HackerOne, and DEFINITELY not through customers.
Hackerone isn't in the wrong, zendesk isn't in the wrong, the author is 100% in the wrong. He broke every single rule imaginable. Now, it's great that he found it and he did need to forward it to ZenDesk, and if he had simply done that I'm sure he would have gotten compensation, but now he caused damage to EVERYONE involved.
I don't care this is going to get downvoted this is actual insanity. HackerOne did it's job correctly, ZenDesk didn't like that the exploit was leaked to people who COULD then abuse it. This is so wrong on every level
RadicalDog@reddit
Grim. It's frustrating because the loss of business was cost by HackerOne, not the teenager. Zendesk are a big enough company that they should pay the bounty and promise to discuss appropriate escalation with HackerOne. Would be cheaper than losing another couple clients, which this could easily do.
klo8@reddit
Yes, agreed
SanityInAnarchy@reddit
No, this is 100% on Zendesk.
First: Maybe H1 should've forwarded it, but they were acting on Zendesk's behalf. You can't absolve responsibility by contracting out to a third party and then blaming the third party.
Second: Here's Zendesk's explanation of why they ultimately paid out $0:
This is disingenuous, of course. The issue had been clearly marked "out of scope" and demoted to "informational". IMO that should be fair game to discuss publicly at that point, yet the researcher only contacted other companies impacted -- companies that were evidently more cooperative, as he ended up earning $50k from Zendesk customers.
At that point, the right move would be, first, for Zendesk to apologize and pay out an appropriate bounty. And second, if they wanted to distance themselves from H1, that was their opportunity -- instead, they essentially doubled down on H1's absurd policy of allowing companies to reject reports, but sit on them anyway.
Bebop3141@reddit
HackerOne rejected it, guy escalated to ZenDesk staff, who also rejected it (it sounds like). A pox on both houses - though I’ll point out that, no matter how badly HackerOne handled it, the bounty policy which excluded this kind of exploit (in principle) was set by ZenDesk.
nemec@reddit
He escalated to a HackerOne mediator, who rejected it.
golgol12@reddit
Is that really a 15 year old? Outside the the headings, it's like GTP wrote it.
RICHUNCLEPENNYBAGS@reddit
I don't think writing this text is beyond the capabilities of a smart high schooler.
golgol12@reddit
I'll reverse my position. I was reading it further and later in the article the grammar and punctuation began to matched what was in the headers. The first couple paragraphs threw me.
Leihd@reddit
Hmm, sounds a bit like if I discover something huge in Zendesk that opens them up to serious legal trouble, they're the last person I should try reporting it to.
Instead I should contact their biggest clients. Not sure how else I'm meant to take their response to what should've been a quick turnaround time even when they acknowledged it, then telling the report to screw off, he's getting nothing.
Empty-Win-5381@reddit
Hahaha EXACTLY. You should just contact their biggest clients and Government regulators while you're at it. They're asking to be grinded
bobsbitchtitz@reddit
This kid is a fucking beast. The fact that ZenDesk fucked up so bad and proably lost > $100k in clients instead of just paying the kids is wild.
Jugales@reddit
If Google can lose all the data of a billion dollar wealth management company, or Cyberstrike can take down millions of corporate computers at once, or the government can have every single person’s SSN stolen, I’m sure Zendesk will be alright. Just gotta pay the PR department a bonus.
KevinCarbonara@reddit
Is there any indication crowdstrike is even going to recover?
iiiinthecomputer@reddit
Sure. It'll be a blip. They'll reband and move on.
KevinCarbonara@reddit
Boeing is necessary to the security of this country. Crowdstrike could be eliminated with a single PR from Microsoft.
oren0@reddit
CRWD stock has regained about 75% of what it lost in the 2 weeks following the outage.
happyscrappy@reddit
I thought Google got all that data back, it just took a while 8 days or something).
bobsbitchtitz@reddit
I'm not saying ZenDesk will collapse but if they lose a couple of clients vs just paying the hacker a paltry sum. Even paying $50k is probably nothing comparatively.
Xenasis@reddit
I'm not in marketing, but to me, $50k is well worth the horrible PR you get for refusing to fix a security vulnerability.
Ancient_Ad5270@reddit
Crowdstrike
Deathisfatal@reddit
Clownstrike
SanityInAnarchy@reddit
For extra fun, https://clownstrike.lol/
Note the same "it's not the crime, it's the coverup" pattern as Zendesk exhibited here. I would never have heard of that website if it hadn't been hit with a series of fraudulent DMCAs and other false positives to try to bully it off the Internet.
Jugales@reddit
I noticed after commenting and wondered how long it would take someone to notice, apparently 3 hours lol
qwak@reddit
I'm telling your customers. Pay me
esperind@reddit
its like when your mom called every game "nintendo".
beaurepair@reddit
The wealth management company deleted their own data, but your point still stands
damondefault@reddit
Did you read this somewhere? It's really the exact opposite of what happened.
Jugales@reddit
Hardly. The Google-built admin tool was created with a default account deletion after 1 year, and their engineers were supposed to manually override that insane default. Bad design.
Worth_Trust_3825@reddit
Not even that. Some companies are just too big to fail.
DJTheLQ@reddit
Their fix is to use RSpamD score and explicitly block Google and Apple reply-to emails.
But neither really prevent this attack? Craft an email that passes the spam filter, attack non Google and Apple SaaS services.
Only something like
support@support.company.com
avoids this.SanityInAnarchy@reddit
Or randomize ticket IDs, so you can't just add yourself to someone else's ticket? That seems like the most obvious fix, and I don't understand why they wouldn't do it.
iiiinthecomputer@reddit
You don't have the randomize the ticket ID, just generate an additional ticket specific token for email references only.
support+ticket1242-GzHkkY@...
KUUUUUUUUUUUUUUUUUUZ@reddit
and im pretty sure that still doesnt prevent an insider attacker from you know, using their own company email to read support tickets.
I.e, join the company as an intern, gain access to data in violation of PoLP policies
vegiimite@reddit
If all that comes out of this a $0 bounty he should consider himself lucky. Writing a script to commit Felony Unauthorized Access is not something I would do.
Sokaron@reddit
Then white-hat is probably not a career for you.
vegiimite@reddit
I think you are naive about what the law is and how little good intentions matter to a US attorney general. I think people on Reddit should remember what happened to Aaron Swartz.
In the US if you don't have consent to log into someone else's computer system that is a violation of the CFAA. Which is what using an exploit to logon to a Zendesk's client's Slack without permission is.
Doesn't matter if you are hunting bug bounties, or even if someone emailed you a user name or password by mistake. If you do not have permission to connect to someone else's system that is illegal.
Honestly, I kind of expected better from r/programming.
ML_DL_RL@reddit
This is crazy since Zendesk is a publicly traded company. Unfortunately, this is not the first time that I see this sort of thing. A lot of this larger companies with old outdated infrastructure, need to get their act together. With the whole AI revolution, it's the perfect time to disrupt some of the dinosaurs out there.
Alborak2@reddit
This was written by a 15 year old? I have coworkers who can't write that well let alone jump through the hoops OP did. I hope OP gets a hell of a lot more than 50k from their future employments.
amsassarp@reddit
man is 15, i’m 10 yrs older and have achieved nothing
throwawaystedaccount@reddit
I'm 20 years older than you and I've done nothing of this scale. Some of us are just normies. It's ok.
Boneraventura@reddit
Probably isnt a redditor
Kok_Nikol@reddit
Awesome job!
And a very distasteful reply from zendesk in the comments, yuck.
Halkcyon@reddit
I refuse to believe a no-pfp, created-today account on GitHub is actually Zendesk.
Kok_Nikol@reddit
Fair enough, but it's kind of bizarre that someone would fake that...
In any case, their official response is equally ugly - https://support.zendesk.com/hc/en-us/articles/8187090244506-Email-user-verification-bug-bounty-report-retrospective
Doctor_McKay@reddit
Bullshit. Zendesk outright rejected the report, which means that they washed their hands of whatever he might do with it in the future. After all, it's "not a vulnerability", so why should he have to stay quiet about it?
nemec@reddit
because he agreed to that policy when he submitted the report. Anyway, "out of scope" doesn't mean "won't be fixed".
harambetidepod@reddit
That was a good read. Fvck Zendesk lol
Tired8281@reddit
Sorry, stiffing me on the bug means that confidentiality is now out of scope. TYVM.
Consistent_Still_668@reddit
I wish I was surprised by this crap.
Skizm@reddit
When zendesk asked him to keep quiet, he should have replied: “sorry, this bug is out of scope”.
Trolann@reddit
That's what I thought. You said I don't have a bug in scope here and I informed, not reported. Then I tell individual companies and now you want to know how I escalated from email spoofing configuration issues, but that is still out of scope.
Tell your paying customers the bug they're reporting is out of scope and thank them for notifying.
victotronics@reddit
Very good read. Hunter fully earned his $50k. Impressive.
"Slack intoduced OAuth login just a few years ago and must have completely forgotten"
That's another big company not coming out of this looking particularly good.
mikkolukas@reddit
Nice job. He will get far. I'm rooting for him! :)
qmunke@reddit
IANAL but I would expect a sudden flurry of GDPR-related complaints against Zendesk - knowingly leaving a vulnerability of this severity open after being notified appears to violate Article 32: https://gdpr-info.eu/art-32-gdpr/
(although they have now remediated the issue, I'm not sure about the potential for back-dating complaints)
Tiquortoo@reddit
GDPR, more broadly, just doesn't actually work how you think it does. There are notifications and remediation periods, etc
Snoron@reddit
I'd be slightly more impressed with ZenDesk after this if they gave this person their WELL DESERVED bounty to show they really take this stuff seriously, as well as it being an admittance that their own response to the initial reports were not appropriate, as that directly led to the later action.
They may or may not have broken the guidelines depending on how you interpret things, but regardless, they simply didn't leave other avenues open.
IF ZenDesk is claiming that a) they didn't want to go any further with that issue, and b) they didn't want it sharing with any other companies... THEN the only conclusion is that they want to leave a bug unpatched without any of their customers knowing about it.