The tale of No Auth Monday
Posted by KevinKeen18@reddit | talesfromtechsupport | View on Reddit | 19 comments
So here we are yet again. I traded my job in the hosting sector for something slightly less… soul sucking in public sector IT. And as many organizations do these days, we rely heavily on Microsoft for a lot of things.
And that sets the stage for today’s tale of horror.
It started simple enough. A few calls from remote workers unable to authenticate into Citrix using Microsoft Authenticator. Annoying, sure, but not exactly catastrophic. We’ve been having enough issues with Microsoft lately that we actually have a backup MFA method for this exact scenario.
So the affected users log in through the backup, we log the tickets, escalate them to the remote work team, and move on with our day. Slack starts filling with the usual Microsoft memes. Business as usual.
Or so we thought.
Suddenly systems start dropping like flies.
First Teams starts screaming that we have no network connection. Which is impressive, considering we work entirely through Citrix. If the internet was actually dead, our entire remote desktop environment would be gone too. So clearly something else was happening.
Then the first colleagues finish calls. Me included.
End call. Move cursor to call system. Click “Done”.
Nothing.
Click again.
Still nothing.
Okay… That’s not great.
Now we can’t change status anymore. People start getting stuck in break states, active call states, all kinds of nonsense. Then internal sites relying on Microsoft authentication begin failing one after another. Shortly after that, external sites stop loading entirely.
At this point everyone starts asking each other:
“Wait… are you guys seeing this too?”
Turns out nobody was hallucinating.
Meanwhile management starts mobilizing. The call queue climbs from 50… to 75… to 100.
Which sucks, but hey, we have a fallback phone system…
And you guessed it, can only be enabled by going to an external site.
So for the next 45 ish minutes we mostly sit there watching the infrastructure equivalent of a medieval city burning down around us.
Eventually, after a long call with telecom, the fallback system finally comes online.
And at that exact moment, our primary phone system decides it’s healthy again.
Of course it does.
And here I am now, two hours later. Still logging tickets. Still telling remote workers:
“Yes, we know Microsoft Authenticator is still broken.”
And so Monday passes by.
Now I sit here on the train ride home preparing tonight’s D&D session to recover mentally from the experience, writing this post as a form of therapy.
So as I’m already in a medieval mood, I leave you with the wisdom this day has granted me:
Should thy systems fail once more,
As oft they have in days before,
Cast Vicious Mockery without fear,
At Microsoft, so all may hear.
CeilingHamster@reddit
One of the most useful things I had at a previous job was the ability to send out a "we know something is down, you don't all need to tell us" to people.
But I assume if you had one, it would have failed.
AStrandedSailor@reddit
You are assuming that: A - users actually read and pay attention to that message and B - that certain users don't think they are special and such messages don't apply to them.
Then, of course, there are the ever so helpful reminders: hey just letting you know we still can't access this. Do you know when it will be fixed?
meitemark@reddit
Those gets a remedy course in basic reading and understanding. The ever so helpful reminder people gets reminded about the remedy course once every hour with snailmail, email and sms until it is taken.
KevinKeen18@reddit (OP)
Yeah we do, and we’d love to. Buuttt ofcourse that’s gated behind our Microsoft login soooo….
NotYourNanny@reddit
I think you have more than one problem.
harrywwc@reddit
nup - just the one - based out of Seattle, WA. :/
Stryker_One@reddit
Technically, Redmond, WA.
harrywwc@reddit
Eh. From the Antipodes same, same ;)
NotYourNanny@reddit
I doubt they decided to put all your eggs in one basket, which is to say, single point of failure with the ability to fix it behind that point.
Though I'm sure they encouraged someone high up in your company to do so.
Inconsequentialish@reddit
But the CEO needs someone to yell at NOWWWWwwwwWWWwwwWWw...!
DaHick@reddit
So I work in Oil & Gas. Formerly in field support. There is nothing less enjoyable than an OIM (the guy who runs the platform) screaming in your ear that this is costing the company over $ 5,000 USD per minute while you are trying to get the platform's power generation online. And I was sitting on a bucket.
Jonathan_the_Nerd@reddit
You've heard of rubber duck debugging. How about... rubber duck blame-taking?
Moneia@reddit
It's one of the good things my ISP does, you call to report a problem and the inbound message is already telling you that yes, they know the service is down, engineers are on the way and there's an estimated fix time of X O'clock.
SwedishMeatballDish@reddit
Just had that happen to me yesterday.
In the time for me to notice my internet was down, and to restart/unplug the modem once, I had a text message from the ISP with an ETA... that got pushed back three separate times, two hours each.
iankel1984@reddit
We had this one day someone left a reply option on a system notification email. About 500 people started replying to all "take me off this list". Server had a shit fit for an hour until someone killed the original email replies.
The4th_Survivor@reddit
In an ideal world that'd be all that's needed. We have that in our place, and during and outage it'll drop the calls from 149 to 150. With everyone being "so I know you said this is down but this is down".
himitsumono@reddit
Have you tried turning it and on again?
Microsoft, that is.
ThunderDwn@reddit
I'm stealing that. Sorry, no takebacks. 🤣
Generic_Placebo42@reddit
I may have to print that and stick it to my momitor at work! Very well done!