Final Update: Microsoft blocked my CPA client's emails the day before the tax deadline

Posted by Lord_Amoux@reddit | sysadmin | View on Reddit | 58 comments

Last post: https://www.reddit.com/r/sysadmin/comments/1sn8c3t/update_microsoft_blocked_my_cpa_clients_emails/

Figured I would make a final update on the situation with Microsoft blocking our client's CPA tenant for a week during the tax deadline.

We continued to ask Microsoft why Huntress or Avanan would cause the tenant to be blocked. They did not know. Instead, they shifted to start asking us to gather a bunch of information for the Exchange Engineering team (further using up more of our time). They wanted :

Two (2) weeks of logs (CSV format) from the Exchange and Defender portals:
Mailflow status report
Threat protection report
Mailflow map
Outbound connector logs
SMTP AUTH clients report
Top sender report (please note any spikes, especially from Postmaster addresses)
A clear summary of findings documented in the case notes, including any anomalies observed in the reports above

At this point I made it clear to support that we weren't going to be the ones to spend our time investigating a tenant that is blocked for reasons they don't even know.

At the same time we had a ticket open with Pax8 who were able to get a Sev A case open with Microsoft. Friday afternoon (4 days after the block began) the tenant was randomly unblocked.

We got a message from Microsoft stating that :

After a thorough review, we confirmed that the tenant was incorrectly classified as abusive due to certain characteristics that matched patterns typically associated with abusive activity. Microsoft uses strict and advanced criteria to identify potentially abusive tenants; however, as some threat actors continue to evolve and blend their activity with normal email traffic, occasional misclassifications can occur.

So after all of that, it was literally a false positive. As we knew from the beginning.

We were called by the Support Engineering Manager apologizing and explained that he reviewed all correspondence between the Exchange team and us, and even acknowledged that "the owning engineers appear to be very unresponsive and at times focused on things unrelated to the issue and caused confusion."

Happy Friday

[-]

MightBeDownstairs@reddit

This same shit happens with Docusign emails all the time too

[-]

ExceptionEX@reddit

Docusign needs to do better about letting customers abuse their service the number of legit from docusign documents that are actually just redirect documents is wild.

I could see them getting banned frequently

[-]

bbbbbthatsfivebees@reddit

It's not always just Docusign. We see tons of phishing come through from random domains that are just exact copies of Docusign emails with the link replaced by a fake 365 login page.

It doesn't usually get blocked because a lot of them come from the domain of an already-trusted companies/users who got compromised, so filtering allows it through.

I'm THIS close to just blocking all emails with "Docusign" in the subject or body. I genuinely wish I could, but I'm not going to because a lot of our clients legitimately use it. Maybe I need to make a training module specifically for emails that look like Docusign...

[-]

ExceptionEX@reddit

We made a rule that blocks all docusign that doesn't come from docusign's domains.

So generally we only see the problem, from actual docusign, we put them in quarantine because some people use docusign to just mean to digitally sign.

[-]

bbbbbthatsfivebees@reddit

We can't put Docusign anything into quarantine because too many of our clients use it daily. But it does seem like a good majority of the phishes that actually hit are impersonating Docusign. I'm surprised that Docusign hasn't cracked down on this and started putting out their own guidance and warnings directed at users. The abuse is BAD BAD. Filtering catches most of it, but the ones that get through aren't easy to detect and could fool even the most savvy users.

I'm going to spend the next week writing and sending out a new training module about Docusign specifically, tell people to check the sender and then call whoever supposedly sent it to double-check that it's actually legitimate.

[-]

ExceptionEX@reddit

We've got people who use it in sister org, we basically have them to external verification.

Generally a call or email, to the party it is from (they disregard the ones unknown)

It isn't perfect, but until docusign gets their act together or someone replaces them, as you said its a pain.

[-]

GroundbreakingCrow80@reddit

PayPal too they don't care at all.

[-]

anonymousITCoward@reddit

That's because docusign is an abused domain... spoofers love it... and adobe too

[-]

mountaindrewtech@reddit

Don't forget about quickbooks! :)

[-]

anonymousITCoward@reddit

oh yea.. those guys too.. i wish i could just blacklist them

[-]

pinkycatcher@reddit

We had an Adobe false positive this morning. Very annoying for my cybersec and the 8 employees affected.

[-]

Arlieth@reddit

The Support Engineering Manager should get you a copy of the CoE findings. You may need to sign an NDA for it though.

[-]

phylter99@reddit

If a lawyer requests the information through a court action then there's no NDA. If I were a business that lost a significant amount of money or customers due to this, I think I might consult a lawyer to ensure it doesn't happen again.

[-]

Arlieth@reddit

Usually these are more informal and given as a courtesy to preserve customer trust.

[-]

wey0402@reddit

Centre of Excellence (CoE)?

[-]

Arlieth@reddit

Correction of Errors.

[-]

KadahCoba@reddit

My guess is that the increased volume from it being tax time again triggered false automated flagging and it only took the first human to look at the reason it was triggered to see that it was false.

But its MS, so it takes an act of god to get MS to have an actual engineer look at the issue and not just reps copy/paste what the dashboard says.

[-]

pandawelch@reddit

Business be businessing - that’s abuse

[-]

Ferretau@reddit

You mean the one engineer with about 100,000 tickets atm due to their use of "AI" to reduce the workload?

[-]

blbd@reddit

I thought the AI was to let the corpos increase the workload per human?

[-]

Ferretau@reddit

No it's to reduce the number of humans required to complete the same amount of work. The Sales guys keep saying it's cheaper to hand it to an AI system and it will do the same job for 1/10th the cost.

[-]

graph_worlok@reddit

Potentially the subject matter as well - Might have a few trigger keywords in there.

[-]

CPAtech@reddit

"the owning engineers appear to be very unresponsive and at times focused on things unrelated to the issue and caused confusion."

In other words, you got the standard Microsoft support experience.

[-]

techtornado@reddit

I’ve been giving Microsoft support a ton of business grade grief for 1.5 years about Direct Send spoofing landing in people’s inboxes

Their “engineer” oh we fixed that, you won’t have any more problems

I send them more and more evidence of spoofs that keep coming despite DS being off

*crickets*

Direct Send Nightmare hits with the fury of a thousand cannons

I
Told
You
So
.gif

[-]

Kurgan_IT@reddit

Gotta love AI and cloud services. I run my own mail server.

[-]

anxiousvater@reddit

I also received a few security incidents from our GSOC, who are a bunch of useless, hopeless folks. They happen to (over) use Defender & not even sure why the event got triggered. There is this lateral movement attack something for KeyVault, I never understood, they never explained (I am very sure they themselves are clueless).

[-]

mapbits@reddit

Leaked secrets are behind many recent high visibility breaches. You need to take this seriously and push them hard for the underlying activity that triggered the alert. Once you have this, only your team will be able to validate whether it's a false positive based on the activity.

It's possible they may also just be reporting on a vulnerability / attack path. Keyvaults by default are not securely configured and can present significant risk to your organization, with common gaps including use of legacy non-RBAC permissions, missing firewall / private link access layer, lack of protection by PIM and phish-resistant MFA, and missing risk-based Conditional Access for workload identities with standing access to them.

Personal rant: Microsoft makes firewall configuration incredibly difficult for access to keyvaults from Power Platform - what appears to be an easy button isn't.

[-]

anxiousvater@reddit

We harden KVs very well, no public access enabled, no VNETs are whitelisted, only private endpoints. On IAM side, only RBAC no vault access policies. The problem isn't that, but this lateral movement blah-blah thingy that should explain the attack path, nothing in there.

False positives are like these fake tiger story, if too many false positives are fired, people won't take the actual incident seriously. And the ones that setup alerts, must know what they are doing rather throwing the burden onto someone who is not aware of the attack path, at least it should be clearly explained.

[-]

badaccount99@reddit

It's not just Microsoft. It's the automation of things and the support people having no clue how that works.

One of the sites my team manages does like 20 million visitors per day. It's a news site. It's gotten blocked by MS, Google, Spectrum, Comcast, and so many more for a few days at a time and then they remove the block with no explanation.

Most likely because they use crappy list providers for bad sites, and if you piss off the right people in a foreign country with a news article they abuse the heck out of those lists.

It's like the old spam email listing services where people could fake complain and then you have to pay them money to get removed.

We're not posting bad content, and we send millions of emails per day too that people subscribe too and bounce/complaint rate is less than 0.01% But that doesn't stop the trolls from trying to get us blocked.

But then also, my team is super guilty about blocking people accessing our site too. OVH, Digital Ocean, and a bunch of countries entirely blocked because Singapore doesn't view ads and wants to crawl us 100 million times per day to steal content for AI. F ByteDance...

It's the internet/cloud You don't control their network. If MS does bad, switch to a different provider. So many alternatives out there that have import features to at least get inbound emails working in minutes.

[-]

dagbrown@reddit

It's like the old spam email listing services where people could fake complain and then you have to pay them money to get removed.

Those haven't gone anywhere.

[-]

badaccount99@reddit

Oh don't I know.

And too many sysadmins just trust them when they shouldn't and do some DNS lookup to block emails in Sendmail or Postfix. And when we want to get removed they ask for a bunch of money.

As a person who ran SMTP servers for a long time back in the days I'm not a fan of the huge email blasts we do. But they all subscribed for it and our complaint rate is so so low.

Spamhaus and others though are not good.

[-]

Michichael@reddit

Ah yes. People relying on AI with life-impacting consequences.

We had a vendor tell us we were using Petabytes of data today and needed a massive true up. We reviewed what they sent. 480 TB datastore? Yeah, 390 GB.

40 TB datastore? Less than a gig. What the fuck. No. Fuck off with your AI bullshit. It sucks, it's useless, and it's actively making everyone's lives worse.

[-]

fresh-dork@reddit

So after all of that, it was literally a false positive. As we knew from the beginning.

and support is unable or doesn't care to see if the tenant is flagged in any way, which would be an obvious first step once any actual mechanical issues are eliminated.

[-]

KieshwaM@reddit

We had something similar a couple of years ago, with our emails being blocked, and same experience as you: a month non stop log providing, every time the ticket got reassigned, new tech wanted all new logs. In the end, turned out their ML anti spam didn't like a couple of links in our signature. Took our CIO getting the MS manager for the country involved to get it resolved.

I'm actually convinced that because support is so bad these days, they aim for log exhaustion, just constantly asking for a new log because they can say "waiting on the customer" for SLA, and just keep doing it until you resolve yourself or get frustrated enough to give up.

[-]

dedjedi@reddit

and you will pay for the privilege!

[-]

jonsteph@reddit

(Possible) AI used to implement a black box to detect threats and empowered to act unilaterally with insufficient reporting. Or, based on experience inside MS Support, the reporting was unavailable to the support engineer.

[-]

ciabattabing16@reddit

Seems that someone was finally able to do the needful

[-]

iamnoone___@reddit

Kindly reverted

[-]

mtgguy999@reddit

Thanks for doing the needful

[-]

people_t@reddit

Let me guess this organizations sends out a whole lot of junk mail that no one wants or needs. Or is sending out stuff that looks somewhat sketchy.

[-]

t0c@reddit

Please read his previous posts so you don’t come to incorrect conclusions.

[-]

Jealous_Crow1346@reddit

Some people are so quick to jump into conclusions.

[-]

Most_Incident_9223@reddit

they want to invent a story where this will never happen to them

[-]

fortune82@reddit

You could make a fun yard game about it

[-]

Lord_Amoux@reddit (OP)

This organization corresponds with individual clients to handle their taxes. They don't send any newsletters or mass emails out.

[-]

RCBing@reddit

A CPA that doesn't send out emails to all former and current customers about tax changes and prep emails? Sure ;) ;)

[-]

vogelke@reddit

Mine doesn't. I get one reminder once a year that it's tax time, and that's it.

[-]

RCBing@reddit

So you get mass email once a year, so a mass email.

[-]

BrentNewland@reddit

You could, you know, click the link in the post, and find out that it's an accounting firm sending out tax emails during tax time.

[-]

Less-Room-9550@reddit

Have they looked into any other potential causes like email volume spikes or content filters triggering?

[-]

Wise-Butterfly-6546@reddit

this one hits. similar pattern with us last year, security tool triggered something on the tenant side and microsoft's classifier flagged us as abusive. took 6 days to clear, partner channel was the only thing that moved it.

the part nobody warns you about: microsoft's abuse classifier reacts to bursty patterns, so any edr or mail security tool that does scan-on-send can look like outbound spam to them. we now whitelist scan traffic against a separate connector so it doesn't blend with normal client mail
synthetic mail check every 10 minutes from a separate tenant to canary mailboxes. if round trip fails twice, pager goes off. caught two near-blocks since, both before clients noticed
for client comms during the outage, prebuilt status page with timestamped updates beat email by a mile. client calmed down once they could refresh a page instead of waiting on tickets
sev a routing tip: open through partner center if you have a csp relationship, the queue is materially faster than the standard tenant path. also reference the case number in every reply, owning engineer changes constantly otherwise
postmortem with the client matters more than the fix. we walked them through what tripped, what we changed, and gave a 30 day fee credit. retained the account, got a referral two months later

glad you got it landed. that final "happy friday" earned.

[-]

RCTID1975@reddit

So if you had a Pax8 contact, why didn't you go straight to them? That's literally the reason to buy licensing from them rather than direct.

Im sure you feel vindicated by the call and them saying it was a false positive, but those things happen, and you didn't help resolve this as quickly as you could have.

Reading your other posts, you were extremely combative which isn't the way to get help.

Additionally, if you're not willing to help provide information, they aren't going to be able to do a RCA, so not only will you never know why the FP occurred, but it's likely to happen again.

[-]

VersaEnthusiast@reddit

Pax8 support has been absolutely fucking useless recently.

[-]

Lord_Amoux@reddit (OP)

The whole issue started with the first ticket we opened having the support rep tell us they were going to unblock the tenant the day it started and they would call us back in about an hour. Fast forward to the next day and we still hadn't received a call back. We emailed back on the ticket and did not get a response. We opened another ticket which got us a call and it was assigned to the same support engineer. They claimed they were not the one that talked to us the day before when they literally had the same name on the ticket. This is where frustration started to build. We provided NDR info and basic email reports (and any other request they made), to which they responded by saying they will run "the command to unblock" once again. Then they emailed us again stating it was Huntress and Avanan causing the tenant block and that we need to uninstall these enterprise applications. We ask why, and then they go back to requesting the same email logs we had already sent. The whole time, 3 days have passed and the tenant cannot email out during the most crucial time.

[-]

Lord_Amoux@reddit (OP)

We opened a ticket with Pax8 the same day it was blocked but their initial response to our ticket was that only Microsoft could unblock tenants and they couldn't do anything. We had to urge them that it wasn't a simple tenant blocked for mass spam issue in order to get them to open up a ticket with Microsoft directly.

[-]