The tale of No Auth Monday

Posted by KevinKeen18@reddit | talesfromtechsupport | View on Reddit | 19 comments

So here we are yet again. I traded my job in the hosting sector for something slightly less… soul sucking in public sector IT. And as many organizations do these days, we rely heavily on Microsoft for a lot of things.

And that sets the stage for today’s tale of horror.

It started simple enough. A few calls from remote workers unable to authenticate into Citrix using Microsoft Authenticator. Annoying, sure, but not exactly catastrophic. We’ve been having enough issues with Microsoft lately that we actually have a backup MFA method for this exact scenario.

So the affected users log in through the backup, we log the tickets, escalate them to the remote work team, and move on with our day. Slack starts filling with the usual Microsoft memes. Business as usual.

Or so we thought.

Suddenly systems start dropping like flies.

First Teams starts screaming that we have no network connection. Which is impressive, considering we work entirely through Citrix. If the internet was actually dead, our entire remote desktop environment would be gone too. So clearly something else was happening.

Then the first colleagues finish calls. Me included.
End call. Move cursor to call system. Click “Done”.

Nothing.

Click again.

Still nothing.

Okay… That’s not great.

Now we can’t change status anymore. People start getting stuck in break states, active call states, all kinds of nonsense. Then internal sites relying on Microsoft authentication begin failing one after another. Shortly after that, external sites stop loading entirely.

At this point everyone starts asking each other:
“Wait… are you guys seeing this too?”
Turns out nobody was hallucinating.

Meanwhile management starts mobilizing. The call queue climbs from 50… to 75… to 100.
Which sucks, but hey, we have a fallback phone system…

And you guessed it, can only be enabled by going to an external site.

So for the next 45 ish minutes we mostly sit there watching the infrastructure equivalent of a medieval city burning down around us.

Eventually, after a long call with telecom, the fallback system finally comes online.
And at that exact moment, our primary phone system decides it’s healthy again.

Of course it does.

And here I am now, two hours later. Still logging tickets. Still telling remote workers:

“Yes, we know Microsoft Authenticator is still broken.”

And so Monday passes by.

Now I sit here on the train ride home preparing tonight’s D&D session to recover mentally from the experience, writing this post as a form of therapy.

So as I’m already in a medieval mood, I leave you with the wisdom this day has granted me:

Should thy systems fail once more,
As oft they have in days before,
Cast Vicious Mockery without fear,
At Microsoft, so all may hear.