Locked out of m365. Clients only IT admin in hospital
Posted by TechnicianAdept1062@reddit | sysadmin | View on Reddit | 105 comments
We were able to regain domain/DNS access with the help of the web developer. I have called the Data Protection Team (DPT) three times today, but the AI customer service system keeps answering and cannot even capture my email address correctly for a callback. They do have my phone number, but it has now been eight hours with no response from an actual human who can take ownership of this case.
We also created a case through our Microsoft Partner Portal, but it is automatically assigned a low priority by default. Can someone please advise on how to reach a human Microsoft support agent from the appropriate team who can address this issue?
Expensive_Plant_9530@reddit
This is why every company should have a break glass account with creds printed out on paper and shoved into a safe or vault for a rainy day.
jfoust2@reddit
You can say that. You'd think Microsoft should enforce it, too.
Last time I posted this comment, it was downvoted and I don't know why. Microsoft is no doubt dealing with many cases like this. It's not fun for them. It costs them money. It causes pain for its customers.
Why doesn't Microsoft scan all the 365 businesses and flag the ones that only have one GA? Force-encourage them all to create at least one break-glass account.
I was bitten by it, with a client I'd set up on 365 many years ago. I generally try to find the best-practices for anything I do, and at the time, I don't remember anything about the need for break-glass. It makes sense in retrospect, of course. Since then, 3FA came along, Authenticator came along, and things change. Sure, I should've reviewed and update their config, but I hadn't.
At some point the business owner GA had added Authenticator to his phone, then treated himself to a new phone at Christmas and didn't worry about migrating Authenticator, and poof, he was locked-out.
There's an endless list of edge-cases and dependencies and ways that these houses-of-cards can come tumbling down. People lose control of email addresses, 2FA devices, passwords, etc. and sometimes it was completely beyond their control. As I pointed out to a client the other day, sure, we need a break-glass account, but why just one? What if we're both together and killed in a crash?
Windows95GOAT@reddit
Then again, the break glass account does nothing to prevent a lockout from a breach.
I am just going to say it; MS support should be better and faster on this front.
Best thing a company can do is inquire now what it would take to get control back through microsoft, collect all that data and then keep that in a "break glass" box in the company safe. Preferably with a break glass account, which again, only helps with fuckups.
platyhooks@reddit
Probably needs to be a FIDO Key with forced MFA for admin accounts in 365.
Expensive_Plant_9530@reddit
Yep, you can handle that with a hardware token or something. Depends on how you want to handle it.
mraztastic@reddit
If you’re a partner you should be contacting Partner Support. Those cases are accessible through the partner portal.
Are you an independent consultant or an actual Microsoft partner? This will make a difference in your path to resolution.
TechnicianAdept1062@reddit (OP)
We are a Microsoft partner , we have contacted them - no response - ticket got auto created at low priority automatically
GrayCalf@reddit
As a Microsoft Partner, you should know better.
There should always be at least two global admins.
ontheroadtonull@reddit
A master and an apprentice.
GrayCalf@reddit
Take my upvote. And may the force be with you.
TechnicianAdept1062@reddit (OP)
I have updated my post, we have been hired to fix this, we didnt manage their IT
CountOfMonkeyCrisco@reddit
Who DID manage their IT?
reseph@reddit
One IT person who is now in the hospital.
CountOfMonkeyCrisco@reddit
Damn, that's a tough one. I've never had to recover from such a situation, but in your shoes, I'd start the process with Microsoft, and then in the meantime be looking for non-technical alternatives. Canvassing other employees to see if one MIGHT have admin credentials, looking to see if there's alternative account that doesn't have MFA on it - something to AROUND the problem while you wait for Microsoft to address it.
Then just keep the customer informed as to what you're doing. Give them detailed updates and realistic expectations, and maybe write up a short report on ways to avoid this kind of problem in the future. Good luck!
iamnoone___@reddit
Hired to fix. Goes to reddit .. ok.
Cultural-Horse-762@reddit
And the post is confusing everyone at that. Half the folks read the title like this is a hospital tenant when, it seems, the precious admin is IN the hospital.
FluffyIrritation@reddit
Oh... I also thought this was about a hospital tenant.
Powerful_Wishbone25@reddit
Year after year I tell myself going on my own will be too difficult. That my skill set isn’t broad enough, or deep enough. That I won’t be able to land clients and deliver.
Then posts like this motivate me to reconsider. Thank you OP.
NerdWhoLikesTrees@reddit
Yeah is OP giving us a cut? Pay up
irioku@reddit
Whatever they paid, they’re overpaying.
splice42@reddit
Surely if you were hired to fix this it's because you have experience and you know how to fix this so there's really no reason you should be asking us how to do the job you were hired for, right?
thortgot@reddit
You were hired to fix and don't know DPA escalation? Be serious
Egon88@reddit
As he mentioned, they didn't manage this, they've been hired to help resolve it.
-TheDoctor@reddit
My boss does not subscribe to this philosophy. He is our only global admin. I have practically begged him to make me a global admin as well (I am our sysadmin), but he flat out refuses. He only wants to rely on admin roles and individual permissions. He even wants to remove his own global admin and rely on break-glass accounts in case of lockouts and such.
If I don't have access or permissions to something in our tenant I have to request it from my boss and wait until he eventually has time to add the appropriate admin role or permissions. Its exhausting sometimes.
When I worked at a University I was a global admin on like day 2, along with our director of infrastructure and our security engineer lmao.
Made_UpWords@reddit
Folks can be cold blooded here sometimes lmfao.
Yeah they fucked up, individually and organizationally, but they're clearly doing everything they can in their capacity to fix it. Can't ask for much more.
TheKosherGenocide@reddit
In my experience, which is about 25 years worth between IT and just other ordinary work, IT people are either the nicest people you've ever met because we deal with so much shit, or the biggest assholes you've ever met because we deal with so much shit.
Sea-Aardvark-756@reddit
I also think the post context is really vague and their idea of "locked out of m365" might just be "the IT admin is out and we need to start doing admin tasks but can't get in as admin" with the rest of the org working. Very unclear.
FastFredNL@reddit
They are not their regular MSP, the customer might not even have a MSP and manages it all on it's own.
mraztastic@reddit
If you’re a Microsoft Partner than you have access to your PDM in the Partner Center. If you're an unmanaged partner, then there are resources for case escalation as long as you opened the case as a partner and not via the individual’s support contract.
You cannot transfer cases between queues. You will need to reopen this if you’re not using a partner support contract.
Grand-Height9907@reddit
This is what happens when Ai is implemented to replace humans
Master-IT-All@reddit
I think you may be expecting this to be resolvable today. It is not. You will be down two weeks.
What you are experiencing is normal.
No one here can help you.
Windows95GOAT@reddit
Not completely. We can give advice and mental support ;)
We sysadmins have to look out for eachoter because we damn wel know no one else will.
25toten@reddit
>no one else will
Rocky_Mountain_Way@reddit
Not true... I can suggest some cheap booze with a high alcohol content to help
BeyondTheHubbleFlow@reddit
Just to add more context you log a ticket with microsoft (as a civilian you may need to spin up a new tenancy to reach them but as a partner you shouldn’t need to) - you go through the v- subcontract, get passed to a second actual microsoft team then eventually get onto the security team as a 3rd point of contact.
The security team gets you to prove legal association with the business owning the tenancy including providing business licenses and photo ID for a legal representative of the company or org then they provide you with access in the form of a break glass account they add typically.
That process typically takes 2 weeks as he said, it can be longer depending on queue times and back and forth on the documentation.
Fragrant-Hamster-325@reddit
I think all this I kind of a fair trade off to prevent some hacker from stealing the tenant.
BrorBlixen@reddit
I would argue that the documentation and identity requirements are needed but passing you around to various departments is process inefficiency and is not enhancing security.
bondguy11@reddit
^^^ Microsoft cases like this take forever to get resolved.
AdvancedAd69420@reddit
28 days. It took us 28 days for Microsoft to do 5 seconds of work.
bazjoe@reddit
What’s your best guess the begging of this though? Hacker or lost password or lost/expire GDAP. Who is the license provider ? You used to have a GA account and now the only viable GA account is to an employee that’s unreachable ?
Drywesi@reddit
The client's IT admin is in the hospital and they had no backup accounts. OP has been hired to get around that.
Cultural-Horse-762@reddit
Weird thread. OP seemingly has tried nothing other than submitting tickets and posts, doesn't seem interested in a technical solution.
disclosure5@reddit
That is the technical solution though. Submit a ticket and wait.
Cultural-Horse-762@reddit
I'd argue that's a pretty administrative solution. A technical one would be to leverage all other controls and access you can wrangle to recover the missing admins accounts, password managers, cached sessions, new tenant then later migration etc. I'm starting to think this is like 10 person tenant for a tiny shop of some sort, the fallout is minimal, and possibly the client impact is barely noticable. Overall none of us can give advice because we still have almost zero context.
Street_Letterhead686@reddit
OP has tried nothing and is all out of ideas
Cultural-Horse-762@reddit
"Hello combined hundreds of years of system experts. I was hired as a tech, can't reach a MS tech. Discuss below.."
Aaaaaaand POST.
felix1429@reddit
You said you were hired exclusively for this situation specifically? Why on earth would you take a job you don't know how to do?
40513786934@reddit
he took the job not because it was easy, but because he though it was easy
BrilliantJob2759@reddit
Probably the owners thinking a Partner could get quicker results than them.
DonOTreply-3477@reddit
Aaaaach, it's just a password and M365, how hard can it be?
DrStalker@reddit
Money.
Duck_Diddler@reddit
TF you want us to do
Jkabaseball@reddit
If the IT admin user is synced from AD, just reset their password and wait for it to sync up to Entra. I know it's not best practice, but worth a shot as other best practices aren't fallowed. Also if they have PIM setup, this would also work.
Cultural-Horse-762@reddit
What exactly is preventing you from logging in?
TechnicianAdept1062@reddit (OP)
We do not have access to the password or MFA, and no recovery email or phone number was set for the owner. The sole IT admin is currently hospitalized, so we have been contacted to perform the recovery process and resolve this.
Windows95GOAT@reddit
So ehm, are we assuming he is not able to speak?
ImDoneForToday2019@reddit
He's finally enjoying some time off.
dnev6784@reddit
Go to the hospital...?
MitochondrianHouse@reddit
I'd run that through the company's legal department. When we have people go out on medical leave, HR has us disable all their accounts because if they do work while being on leave it gets messy.
ImDoneForToday2019@reddit
Plus if they do work while under sedation, that gets real messy also.
SinHazzard@reddit
What is the highest privileged account you have access to?
rodeengel@reddit
If you’re in the US hate to say it but that’s not HIPAA compliant. Their Privacy Officer, a designated position under HIPAA, should be notified of all of this and part of the recovery team due to the nature of business. If you’re not in the US, then you might want to check your local laws.
The reason I say the Privacy Officer is because it’s part of their duties to make sure the business is HIPAA compliant and an oversight like this is huge and would not pass an audit. HHS might also have something to say but if you report this to the Privacy Officer you have done your part.
LadonLegend@reddit
...the client isn't a hospital, the IT admin was hospitalized.
wonderwall879@reddit
and on top of that, none of this has anything to do with HIPAA. This wasn't even a security breach, they're just locked out lol. Privacy Officer would laugh you out their office and say freaking fix it, i dont care, not my problem.
Cultural-Horse-762@reddit
Can you recover/take over any local aspects of possibly his domain user or workstation, maybe a password manager? Other than waiting for M$ to fix it all, you'd have to get creative.
TechnicianAdept1062@reddit (OP)
unfortunately no
operativekiwi@reddit
Unfortunately no? Im very sure you can find a hacky way to solve this... you need to give more context here
rybl@reddit
Tell us more about this. If you can get into his PC that has cached credentials that could be a quick path to recover.
You don't have access to a PC or laptop that he worked from? If you do, was it joined to a local domain or Entra joined?
Cultural-Horse-762@reddit
What environments do you have management of besides domain/DNS? If you have DNS, then you can receive emails to any address at the company domain.
tango_one_six@reddit
You need to set expectations now with your customer that expected resolution will take at least 2 weeks. MSFT takes cases like these both very seriously and not seriously, pointing to the policy that M365 service (like any cloud service) is a shared responsibility - customers want MSFT to keep their data private even from MSFT itself, and the tradeoff is that customers retain sole responsibility and access to their data.
insaneturbo132@reddit
What’s actually wrong that you need access to it to begin with?
Barrerayy@reddit
My guy if you are in a situation where calling the Data Protection Team is your only choice then you are fucked for the lack of a better word.
You'll be down for 2-3 weeks.
crystalbruise@reddit
At this point I’d escalate through every channel you have: Partner portal, distributor/account rep, and any Microsoft partner contacts directly. Mark it as business-critical and emphasize healthcare impact. Cases often move faster once risk and urgency are made crystal clear to a human.
M4tchB0X3r@reddit
Glass break account lesson.
DerpyMcDerpFaceII@reddit
TIme to visit the hospital or that dude may not have a job to go back too
RCG73@reddit
Not to be indelicate but this sounds more like a “hit by a bus scenario”.
weHaveThoughts@reddit
We phrase it, “won the lottery and said fk off to the company” now.
iwinsallthethings@reddit
He’s in the hospital. Bus more likely.
Catsrules@reddit
They could have won the lottery and got hit by the bus.
RCG73@reddit
This would be my luck Yay im rich. Ohhh now I can pay my medical bills. Fuck this timeline
SystemGardener@reddit
How you can tell this is OPs first rodeo sadly. It’s going to be at least a week.
2Tech2Tech@reddit
A1 license is going away, moving to A3 license
Level_Working9664@reddit
Call the person at Microsoft who accepts payment in finance.
Explain the problem and do not get off the phone until they put you in front of a human.
It's the sad reality that behaviour is like. This are the only way you get anything out of Microsoft.
If enough people do this with finance at Microsoft then they will have something internally to say about it.
Geminii27@reddit
Spend tens of millions of dollars a year on Microsoft products and have a specific vendor rep assigned to you.
404error___@reddit
Just ask Copilot!
Danowolf@reddit
Who is your O355 reseller? Hopefully you have one. Once you fix this nightmare, engage a new nightmare with bigger payoff. Move your O365 account to Appriver. And highly consider getting approval for Mimecast. If you had Mimecast your mail may have still flowed.
MorallyDeplorable@reddit
LPT: Microsoft gives zero fucks about problems that are entirely due to the client being a dumbass.
everettmarm@reddit
If they have access to their DNS you could spin up a temp tenant and get them some email dialtone and comms. Post-incident you’d have some lifting to do to re-integrate it all somehow but in a crisis that’s the kind of thing you can pitch.
InevitableOk5017@reddit
Reset the password and mfa through proper ways and boom.
HowardRabb@reddit
Make sure the client is prepared for around a 2 week disruption. They could be locked out that long, or longer. Beat of luck to you, this is a terrible situation to be in.
sryan2k1@reddit
2 weeks is typical for the DPT.
AcidBuuurn@reddit
Bring a laptop and a stack of money to the hospital.
dnev6784@reddit
This
amcco1@reddit
It will be a week+ before you get access.
At that point, just go buy a different domain, create a new tenant. Assuming small company, so just give them a couple of accounts with licenses and hopefully they have like their excel and word docs saved locally on server or machine. Maybe they can scrape by and get some work done.
weHaveThoughts@reddit
As a Partner you can open a Priority “Severity A” ticket. Depending on your partner level you will get a call back between 1-4 hours. But don’t freaking hang up. When you open a Severity A ticket you have to work non-stop on it until it is resolved. Meaning make sure you call in backup.
Oh yeah you must pay for the benefits before calling.
Wise-Butterfly-6546@reddit
been in almost this exact situation with a healthcare client. few things that helped:
if you have Global Admin or at least a Partner relationship in Partner Center, you can submit a support request on behalf of the tenant directly. don't go through normal consumer support channels, use the Partner Center admin link. that gets you to actual engineers, not the AI phone tree.
if the only GA account is the one you're locked out of, and you can prove domain ownership (DNS TXT record), Microsoft can do an admin takeover. it's called "internal admin takeover" in their docs. takes about 24-48 hours but it works.
for the future: always set up at least two break glass accounts with Global Admin. no MFA (or FIDO key in a safe), no conditional access policies, stupid long passwords stored offline. name them something like BreakGlass01@domain.com. I've seen too many orgs with a single admin account that becomes a single point of failure exactly like this.
also check if the original admin set up any emergency access through Entra. some tenants have it configured and nobody remembers.
the Microsoft Partner support line (not regular support) can also escalate faster if you explain it's a healthcare org with patient data access implications. HIPAA urgency tends to move things along.
Low-Prize-9289@reddit
Do you have GDAP setup?
MSP_1010@reddit
Do you have a GDAP relationship?
Ill-Barracuda9031@reddit
Do you have any remote management tools to login to his pc?
TxTechnician@reddit
They don't exist, it's all CoPilot now.
Ok, something that may just be a typo. How do you have a Microsoft Partner Portal? That's something which exists for resellers of M365 products and devs.
You may get your licenses from a vendor (not direct via MS). And if that is the case they may have Delegated Access and might be able to help get the global admin.
Triairius@reddit
This is a tough lesson in single points of failure.
TxTechnician@reddit
Holy fuck that headline.
matt95110@reddit
I think this is one of those situations where you are completely screwed.
alpha417@reddit
With no fallback or prior planning put into this.
disclosure5@reddit
You don't get callbacks in eight hours on this type of work. It doesn't mean they haven't captured your ticket, it just means you're still in a queue. I know it feels frustrating and that you need your business recovered, but these things only happen after multiple missteps.
The fact the domain was in the hands of a web developer is a pretty good indication there is some significant uplifting to do in this client's management.