Tough situation at work
Posted by dwboutTh4t@reddit | sysadmin | View on Reddit | 32 comments
Hi everyone,
im currently preparing for an interview and it's got me thinking about those "day from hell" scenarios we all run into as service desk.
I would love to hear some of your stories. When things went sideways, how did you actually handle it? What steps did you take to fix the mess and how did it all turn out in the end?
I know this sounds like a classic interview but im genuinely interested in the "boots on the ground" perspective and how you guys navigate the chaos.
not_so_wierd@reddit
I needed to update the department name for about 20 users in our AD because their group had been renamed.
Didn't want to do it manually for all 20 so I ran the change via a PowerShell command. Done, and then I moved on to the next ticket.
What I didn't see was that I had messed up and applied the change to EVERYONE in the organization (about 2000 users). We had several scripts and policies that apply based on your Department, so shortly after tickets started flooding in. People could no longer access the tools/sites they needed, they got included in the wrong mailing lists, etc. etc.
Fortunately, I had an export of everyone's department from before the change, so it was easy enough to revert. But people were seriously pissed for a while.
rubmahbelly@reddit
Holy shit. Nightmare fuel. Good for you that you could fix it so fast.
I believe there is a „what if“ function in PS to do a dry test run.
dwboutTh4t@reddit (OP)
i didn't know this function exists in PS. i will try and use this function Thank you!!!
rubmahbelly@reddit
https://amzn.eu/d/0drrOsYM
AdvancedAd69420@reddit
Changed a conditional access rule and locked the entire tenant out for 35 days. Ended up giving the entire company access to barracuda email gateway to give them access to their emails for that entire time. Luckily I have some good people I work with and we made the best of it. Literally the worst mistake I have ever made in my professional career. Fuck Microsoft and Microsoft Support. It took them 35 days to verify we owned the tenant and 5 seconds to fix the lockout.
dwboutTh4t@reddit (OP)
what was that conditional access rule? was it like the "Named Location" rule?? Im really curious how did that happen
AdvancedAd69420@reddit
We have a CA rule for USA only signins. We had a employee going out of the country and Iade another CA rule to allow login for that country. For whatever reason, obviously my mistake our break glass account was not excluded. The 2 rules did not stack. And the end result was we were completely locked out of our tenant. It was a nightmare.
Reedy_Whisper_45@reddit
Ransomware. I came in in the morning and everyone told me their system was acting funny. Multiple systems.
I couldn't log in. My admin account couldn't log in. Our backup admin account (that we never use!) couldn't log in. Took me about 20 minutes to figure out. Took me 2 weeks to get back to operating functionality and another 2 weeks to get rid of the extra fences I had installed to protect us while rebuilding.
It was one thing at a time, and I had a manager who was my gatekeeper. People went to HER, not me, to ask for things. She approached me as needed, and listened when I had things to pass out.
The keys:
I could probably go on. But they all come down to: Do the job.
dwboutTh4t@reddit (OP)
i noticed that most people who replied here always say that their manager really helped them a lot during those incidents. I think that one has a big impact as well so we don't panic while it is happening. anyway.. thanks for sharing!!
Simple-Kaleidoscope4@reddit
You want an example with what you did to make sure it never happes again.
A change that went sideways that caused an outage and you automated the steps next time.
The sql update where you missed the where statement
dwboutTh4t@reddit (OP)
that's a really good guideline for preparing for an interview. thank you!
Glass_Call982@reddit
I deleted a financial controller's active directory account, which in turn deleted their exchange mailbox. Had meant to delete the one below them in the list. I did call them right away and they were understanding but it was embarrassing for me and I'll never do that again. They had random issues with their email for weeks afterwards.
witterquick@reddit
We would generally disable an account then go through a quarterly clear out of disabled accounts. Long enough to act as a scream test should the wrong account be disabled, or if it's a genuine leaver whose account is tied to some kind of service
dwboutTh4t@reddit (OP)
when you said clear out disabled accounts.. do you delete them? we usually convert the disabled accounts to shared mailbox to keep the data
Glass_Call982@reddit
Yeah, back then I was a mid level admin at a small MSP. This was SBS 2003 server. No processes or anything lol.
TheGenericUser0815@reddit
Keep in mind mailboxes which were created as resources implicitly create a deacitvated user account. You don't want to delete those.
witterquick@reddit
Are those accounts created as disabled? I didn't think they were, thought they came enabled but with a system generated password - off today but I'll need to test tomorrow
WizardsOfXanthus@reddit
Wait! Are you looking for answers to spit out as your OWN during the interview, when they inevitably ask you the same question?
dwboutTh4t@reddit (OP)
Hi!! no no noo. just really curious about other people's experiences. and yes i did post this in other subs because there are some subs that wont let me allow to post and i dont know how i can get more karma. I dont always use reddit and just really reading stuff in here when i want to or if i accidentally open reddit 😅
TopHat84@reddit
That was my thinking as a well. Either that or the account it looking to feed some AI algorithm.
WizardsOfXanthus@reddit
Good point. Look at OPs karma and other posts. Same question asked in other subs.
Creative-Package6213@reddit
No chatGPT I will not feed your algorithm...
Impossible_IT@reddit
Claude, is that you? No, I’m your copilot!
EggElectrical669@reddit
Some days you just have to stay calm and deal with one problem at a time.. stepping back and fixing the worst stuff first usually helps. most of the time, things end up working out even if it feels crazy at the moment.
PositiveBubbles@reddit
That was me all week. Got in the shower tonight and just let it all out.
I feel better that I'm only tired after complex tasks and not emotional anymore which is growth.
No point worrying about what is outside our control
rubmahbelly@reddit
Biggest F up was a SCCM deployment which force closed apps. During business hours. I could stop the deployment after the first 10 or so tickets and corrected the job.
My luck was it was late in the day so not too many people got angry.
BadSausageFactory@reddit
I had a full day last week making up stories for my AI channel
poizone68@reddit
hopefully with wonderful narration that instead of saying "I had a job in the early 2000s" says "I had a job in the early twothousand seconds" :)
not_so_wierd@reddit
I was staying late one night to install an update to a production critical system. Should be done around 22:00.
At 23:00 I've managed to download and package it. Start pushing to the workstations. At 01:00 the results are in - 197 out of 200 devices failed to install.
I tried to push again, but it kept failing, so around 02:00 I had to give up and break out the USB drive. The next 12 hours consisted of me going to each device, signing in, and installing from USB. Half the devices were still not working by 8 when the workers showed up. Finally finished at 15:00 - about 36 hours after I came to work.
And of course - the week after I got called into HR for falsifying my time-sheet. "You couldn't have worked that many hours. It's not allowed".
"Don't I know it. But check the cameras, check the access card logs, you'll see I was here. Then ask the site director if he'll approve the overtime or if he'd prefer that no one can work".
GYuGYu_jol@reddit
once i created a list of VMs the agents were using. the practice was to comment the ones to be excluded as they had agents on them, save, then do a shutdown -r -t 0 on the list. this time i forgot to save, and the whole end-user service center went down. luckily it was nightshift back in 2010 so only 2 agents were affected and no calls were coming in during the reboot
The-Sys-Admin@reddit
Ideally with the help of a good manager. Spent a week with a recovery team to bring all our VMs/domain back from a ransomware attack. I was doing the needful and my manager was running interference on anyone who came into the response room. Whether they were an exec who needed their special tool next (we have a list of priority servers, yours is a tier 2, sir) or my own CIO who was begging me not to quit in the middle of this, as the only sys admin.
Im not the kind of person to bail when things get rough, these people needed my help and I had the most knowledge of our infrastructure, there was no one else. Got a nice bonus check for the extra hours I had to put in (120 hours in 10 days), but really my manager stepping it to just let me work really helped.
Of course later on they said "I've seen what you're capable of during the ransomware attack, that's the standard in going to hold you to." Like that wasn't me in crisis management mode, missing out on my family, and being stressed to the max for the whole time.
Don't work there now, but I did do my best to give the next guy a decent lay of the land, unlike my predecessors.
ConstructionSafe2814@reddit
I restarted our file server service and noticed everyone got up at the same time asking me if there was something wrong.
I learned that it's not transparent for end users and did not do that again.