What’s one “small” IT task that somehow always turns into a full-day disaster?
Posted by tresorrarereviews@reddit | sysadmin | View on Reddit | 122 comments
I swear every time I think “this will just take 10 minutes,” it ends up breaking something completely unrelated.
Last week it was a simple update… somehow turned into troubleshooting auth issues, random service failures, and a user insisting it’s “been slow for months.”
At this point I don’t trust anything labeled “quick fix.”
What’s your cursed task?
muscclinad@reddit
For me, it's always DNS changes; what should be a simple record update inevitably leads to propagation delays and inexplicable outages. I've been looking into solutions that automate some of the common tier 1 tasks to avoid these kinds of issues, and Risotto seems promising for automating some of the more common access and troubleshooting requests.
Winter_Engineer2163@reddit
The "Simple" SSL Certificate Renewal
On paper, it’s a file swap. In reality? ->The private key is missing. -> The intermediate chain is broken. -> You discover a legacy service pinned to the old thumbprint. -> Suddenly, nothing can talk to the database because of a TLS mismatch.
smartguy_x@reddit
This is painfully relatable. Half the chaos comes from discovering these issues *during* the renewal instead of weeks before. Tracking cert expiries, thumbprint dependencies, and chain details proactively makes a huge difference. We built Tokentimer (tokentimer.ch) specifically for this, it monitors certificates, tokens, and secrets across environments and alerts you early so the renewal is actually just a file swap. Might save a few full-day disasters.
FalconDriver85@reddit
Please, don’t make me think about it… we are in the process of redoing our PKI infra and I wanna start crying just now before we even start discussing it.
ansibleloop@reddit
Full chain is a bitch because it's easy to miss
Windows machines may says "yep, looks good to me" but apps will go "where tf is the intermediate cert?"
Icuivan@reddit
Printers
Jofuzz@reddit
Set up new network and infrastructure for sister company, then go to set up copier email with oauth 2.0. Oauth works fine, token works great, but emails don't send and copier fails on its built-in SMTP check with a network error.
I think "maybe it's a quirk with ms365". (Picked up 365 a few weeks ago for sister company because I've been fully on-prem my entire career). A couple of days later tenant is configured and have added conditional access policies, made changes to sharepoint, exchange, b2b guest users, etc etc etc. every time I turn a corner I'm learning something new with 365 on top of my regular daily duties.
Get done, go back to copier, still fails SMTP check. I was already suspicious that it wasn't an auth error and was a general SMTP can't connect error but I had to configure the other stuff anyways. This confirmed it.
I check that the firewall isn't the issue, other devices can contact smtp and HVEsmtp servers. Still fails auth. Update firmware and it still fails auth. Scour the settings, checking TLS and everything else, still fails.
I'm leaving out some troubleshooting and research but I spent some time on it.
It's been a week since I set out to add email to the copier. Gonna jam on it more tomorrow morning. At least the 365 tenant is mostly set up now lol.
Elensea@reddit
Make sure the authenticated smtp is allowed on the user level in 365.
Jofuzz@reddit
Thanks for the trip, tried that last week. There's an "Enable SMTP Authentication" checkbox on the copier as well, all enabled.
wazza_the_rockdog@reddit
SMTP2GO or similar will be far quicker, cheaper than a weeks worth of wages, and honestly worth it for your sanity.
juliejujube@reddit
I spent like 45 mins troubleshooting a printer that was connected to the network with no internet access once. We had wonky dhcp issues with items stealing each others ip address so i thought it was that. It was actually the mini switch that it was plugged into got unplugged from the wall. 😭🤣. i felt so dumb
KFJ943@reddit
My old workplace had this incredibly obscure label printing setup - I got a really basic sounding ticket that sounded like I just had to go on location and change a font size to get something to fit.
Nope, it involved installing some ancient homebrew firmware on the printer that a developer who'd quit four years ago made, and never documented. These things were absolutely business critical. If you did any one step in the wrong order, you had to start from the absolute beginning.
Fun times! At least the people working at that site were incredibly nice.
Responsible-Slide-95@reddit
Shhhh.
Im in the middle of a company wide printer refresh. 300+ machines scattered over 200+ locations. We're doing the furst big sure today
Perfect_Designer4885@reddit
May the force be with you!
ClozetSkeleton@reddit
May God have mercy on your soul.
reddit_pug@reddit
I have a small computer shop. If you buy a computer from us, we include transferring your stuff from your old computer to your new one in the purchase price, assuming you bring your old computer into the shop for us to do it there. There. Every so often we have a customer that says something like " oh it would just be easier if you did it here" (at their home or business). We're happy to do that at our normal hourly rate, but sometimes the file copying just drags on. We try to give the customer a heads up that that could happen, but sometimes they just don't listen...
Always_FallingAsleep@reddit
I feel your pain. This is one of those assumed to be a 5 minute jobs. Because copying hundreds of Gigs of data is not time consuming obviously. As well as helping to find software, passwords, accounts from their old machine.
I flat out refuse to do that type of thing on-site nowadays. The including it in the price in the shop if you buy from us is a good incentive. Got to love those people who'll go to a chain store and expect you to do it all for nothing, even show up unannounced. Like here you go. Please transfer from my old system to my new one. That won't take long being that it's so easy. The salesperson in the chain store reassured me that it was.
Always_FallingAsleep@reddit
There are no "small jobs" or tasks when it comes to IT.
Or the moment you hear those words: This will only take 5 minutes or 10 minutes ought to trigger a blaring warning. And the inverse is true also. The ones expected to take most of the day are typically the easy ones. Well they aren't the disaster scenario anyhow.
ourmet@reddit
Updating certificates.
You never get them all and something obscure will throw a fit.
AforAnonymous@reddit
Extra credit version: Offline Root CA CRL rotation.
On steroids: Someone forgot to mark the CRL expiration in the calender.
Character_Deal9259@reddit
My favorite situation was a server that had it certs updated, and so the ticket for that was closed.
Of course, 2 hours later, the server shit itself and went offline. Tech who was working on that ticket restored the server to the last known good backup. Without the certs of course.
Fast forward a week later and everything is broken because the certs expired and people are running around trying to get them replaced. Of course, the guy who replaced them originally is off that day, and unavailable to talk to.
It was a fun day...
tanzWestyy@reddit
GLHF with the new certificate changes coming soon. 100 days? 40 days? Yeww
Ummgh23@reddit
Dont forget the 10 days for domain validation!
phungus1138@reddit
Wildcard certs for when you want to break several things at once.
Legionof1@reddit
Just make a bunch of rolling wildcards!
Ummgh23@reddit
One is plenty :‘)
Ummgh23@reddit
Nono the trick is to just use one wildcard cert for the top level domain for everything
CorvusTheDev@reddit
Please. It's Monday. I want to sleep tonight.
fearless-fossa@reddit
I've once had the same guy who gave me a wildcard cert instead of service-specific ones as I had requested revoke that wildcard cert about two weeks later without notice because "wildcards are inherently insecure and shouldn't be used"
That was a fun morning looking into the browser and half of the monitored services being red.
Apprehensive_Win7049@reddit
I feel seen.
Ummgh23@reddit
Oh god thanks for reminding me, I have to do that this week.
And then proceed to finally find the time to implement acme before certs get to the 47-day lifetime.
wazza_the_rockdog@reddit
Will depend on what needs the certificates and how nicely they can be scripted in, but I find in some cases it's actually quicker to implement certs via acme than it is to do so manually. May be worth trying to move some things to acme generated certs instead of manually replacing them.
Ummgh23@reddit
I‘ll have to set up acme.sh along with a proper, centralized, reverse proxy and acme-dns for the DNS-01 challenges (we don't have the option to use a direct API for our dns entries, so it'll have to be done via cname).
A lot of our stuff runs on IIS, but theres also some apache on windows as well as linux servers. Theres servers with their own IIS reverse proxy for authentication too. So the time will be spent taking inventory, implementing the (docker) infra, documenting and finding the best ways to implement acme for each app.
And more importantly I have to learn how acme and all the tools work. That's why I won't just hastily set something up. We don't have any experience with these processes in our team, so I'll use this to build knowledge and document it well enough so the others can use it too. I won't use it in production if It's not rock-solid yet.
The missing experience in our team is also the main reason I'll probably use NPM instead of something more involved like traefik. I'd be happy to learn it but the others.. not so much
m0rd0rian@reddit
I don't get it. It is easy to automate, especially in the modern environment.
_MusicJunkie@reddit
Man, the time I've spent automating the non modern parts of environments... Unfortunately it is not always easy.
lethargy86@reddit
Certs are used for a lot more than typical web servers that have well-trodden ways to automated cert renewals.
The problem is installing them into arcane COTS products and so forth on an automated basis.
Doable but each product needs its own script that you have to make after reverse-engineering what the software does to install it after you do it manually.
It's kinda fun but a pain in the ass at the same time, because the vendors never make it simple or easy.
m0rd0rian@reddit
Maybe I'm a bit shortsighted as devops in gamedev and softdev but still I haven't had problems with certs for more than 5 years, even with a crap like Fortigate or Cisco
FalconDriver85@reddit
How many SAP systems do you have to manage? IoT devices that use client certificates? There are a couple (actually more) cases when you can’t just monitor the certificates or slap them inside a KeyVault and call it a day…
FirstStaff4124@reddit
What makes you think customers run modern environments?
jspears357@reddit
Like Cisco UCCX servers?
m0rd0rian@reddit
skill issue
ThoranFe@reddit
time issue
TheDawiWhisperer@reddit
"everything should accept a PFX" is a hill i'm totally willing to die on
oppositetoup@reddit
I've just spent a week updating a wildcard. I've come in as a contractor and nothing was documented after the only guy left, and none of the application guys had any idea where they were in use. That was fun...
delicate_elise@reddit
Anything that involves a ticket with Microsoft.
Twist_and_pull@reddit
OP said full day disaster, not full month disaster.
ThePodd222@reddit
A month! Who's getting their Microsoft tickets resolved that quickly?!
GremlinNZ@reddit
Well look at you and your lofty goals!
Closed one ticket months later. Microsoft reaches out, how did you fix it? Uh... We just gave up...
Ganjanium@reddit
This is my experience all three of the times I’ve gone to MS with a problem
Master_Direction8860@reddit
😂😆😂
aenae@reddit
I need to buy lottery tickets. I had an issue with Microsoft rejecting our mail after using a new IP for our mailserver (and not the default reputation/rate limits, but a direct block, so no way to raise reputation, i couldn't even send a single mail).
I contacted support. They not only read the emails and asked some good questions, they also lifted the ban and raised the reputation of our new IP to the same level as our old IP. And they answered every mail within an hour. I did not expect that. I guess the alternative was over 100.000 complaining users, as we send that many mails each day to *.outlook.com.
Miserable_Pear_6940@reddit
I feel seen. I feel heard.
l_ju1c3_l@reddit
AT&T router swap/public IP change. They will mess it up, I guarantee it.
nkwell@reddit
This is pretty much all of internal IT.
"Well, that should do it."
15 mins later
"Oh, that's weird. Oh yeah, that one thing, lemme fix that"
15 mins later
"Brenda from accounting can't do something"
VA_Network_Nerd@reddit
Sorry, it seems this comment or thread has violated a sub-reddit rule and has been removed by a moderator.
Inappropriate use of, or expectation of the Community.
If you wish to appeal this action please don't hesitate to message the moderation team.
Defiant-Chip6513@reddit
Last minute new user set up. Admin somehow always forgets to give IT a heads up (at least 2 days notice).
deathrow_99@reddit
Logging into work in the morning. Always trouble!
Adventurous-Cat8847@reddit
quick permission or config tweaks - always spiral into chaos.
ryoko227@reddit
The best part is when you revert it to a previously working version to try and sort out what went wrong and then that doesn't work now either...
gardnerlabs@reddit
Driver/firmware updates
ImLookingatU@reddit
Switch firmware updates. I have never in my 25 years of IT not had at least ONE switch shitting the bed after an update, cisco, juniper, dell, hp, etc... it doesnt matter.
ReputationNo8889@reddit
We have had a vmware cluster switch that was updated by our vendor completely bricking itself taking the whole cluster down. Took a couple of minutes until everything was running with the secondary switch but they could not bring the original one back to live. after 4 hours they installed a new one, at our expense of course.
thatpaulbloke@reddit
Have an external storage for switch configuration, export the current running config to the external store, install the firmware update and test thoroughly to make sure that the config is still in place and the update completed successfully.
The gods laugh at your hubris and the switch bricks itself completely.
delicate_elise@reddit
This is usually the case if you never update them to begin with. If you keep current, the changes between versions are minor and very unlikely to cause issues.
Enough-Collection-98@reddit
I don’t think that’s how firmware flashing works. You might have separate sectors in flash but you still flash the whole sector, not just the individual bits that might be different between versions.
sdoorex@reddit
It’s not the number of sectors that change, it’s how the settings are applied. Frequently applying firmware as releases occur means that changes can usually be handled by the update process automatically instead of becoming a heading change of you jump too many versions at once. It’s why many vendors have a published firmware upgrade pathway for catching up.
oxmix74@reddit
Also, the vendor tests updates from one version to the next, but they generally don't test updates from an arbitrary early version to current.
RockNRollNBluesNJazz@reddit
Driver updates can be from hell if they've changed some default settings and/or don't carry old settings over. The world has moved from the 90's Plug'n'Pray to the 20's Plug'n'StillPray.
But firmware updates are a different animal altogether. I actually enjoy doing them, because I can do them in peace. Nobody dares to intervene, because I tell them it will take 3 hours. I sit down, relax and drink my coffee. Job is finished in 5 minutes, but I don't put the system online before I'm ready to finish my happy extended break. When I tell everyone I finished only after 2 hours, I get a welcome of a hero. Sweet. I wish I could do more firmware updates. Yes, I am a twisted soul in many ways.
ryoko227@reddit
A reimage that should take 20 minutes, but doesn't image correctly. Then when it does, turns out the image was out of date and needs an update. Typical Windows, the PC locks up while updating and bricks the boot loader. At the end of the day now, you reimage it one more time, and magically as though it was only intended to waste your whole day, and it works without issue.
honey_badger010@reddit
Anything on a Friday, just after lunch.
rootpl@reddit
Every, single, fucking, time.
Particular-Way8801@reddit
Replacing a UPS on a client site on a friday at 17.00 , 2 hours from home, should have been easy peasy, remove old UPS, attach new UPS to switch, enjoy... ? no
switch restart in loop, rolled back to previous UPS
All good except there was no more configuration, if it was not the L3 switch doing all the routing for said client.
No issues, let's grab the backup that the previous consultant had done of the configuration,
said backup consisted of the banner and name of the switch.
Had a fun two hours rebuilding the configuration from memory (like 6/7 vlans) a bit of try and error.
That is the day that I swear to make backups. made a rancid to back up all configurations over the dozens of customer we had.
Fun story, after i left that company (around 10 months or so) they called me saying that they had an issue with a dead switch, and it was changed under warranty(yeah HP/Aruba), but they could not find the backup on the rancid.
Agreed to a phone call to help them.
Mind you, I had the RANCID_TEST_DO_NOT_USE VM that i had powered down before leaving and the RANCID_PROD_DO_NOT_TURN_OFF VM on with multiple email explaining the in's and out (I used the test one to play with perl scripts to try without touching the prod machine).
What they did :
Turned off the "DO_NOT_TURN_OFF" VM, removed the autostart,
started the other VM and complained there was nothing on it....
Fix was easy, start up the right vm, check the logs to see it had been powered down 4 months after i left,
got back the backup file, saved the day, wrote an email explaining what happened, still waiting for some appreciation token (probably will never receive one)
bellysavalis@reddit
Read Only / No Change Fridays...
tuxsmouf@reddit
I still got "PTSD" after a switch loss in a stack of 4 a friday afternoon at 3pm.
I was in "full stress" mode when a secretary (known to not be friendly) came because she couldn't print her document. I explained her quietly to resolve her problem I had to replace the dead switch.
She insisted for a quick solution...I simply said I had no quick solution and told her to let me do my job in a not kind way.
She left not happy and could focus again.
That was 3 years ago but it's still fresh in my memory :).
StiffAssedBrit@reddit
We all have that customer who logs a fault, the calls every 10 minutes to ask how it's going. My response. " It isn't being worked on because my phone keeps ringing!"
picturemeImperfect@reddit
Phone systems
UptimeNull@reddit
Changed a password. Had not done that in a while. Waited and waited for hybrid connector to connect. Ended up delta syncing it ffs.
demalo@reddit
AD is supposed to replicate password changes immediately. Entra connect always waits for the scheduled sync. It can be a crap shoot sometimes.
Sufficient_Duck_8051@reddit
Gotta be the printers or anything that has to do with them.
Zaiakusin@reddit
Printers are the devil.
Extra-Organization-6@reddit
printer driver updates. every single time. just swap the driver, five minutes. then its port conflicts, spooler crashes, one tray stops feeding, and suddenly youre reimaging the print server at 6pm on a friday.
Extra-Organization-6@reddit
printer driver updates. every single time. 'just swap the driver, five minutes.' then it's port conflicts, spooler crashes, one tray stops feeding, and suddenly you're reimaging the print server at 6pm on a friday.
Razee4@reddit
This isnt happening only in IT my friend. As an ex-sys admin, ex-infrostracture builder and now medical service technician - the task you think will be a breeze - never is ;).
Friendly_Guy3@reddit
I learned, if someone is deploying certs in the wrong store , my sccm will throw a fit . Not recommended.
KlausBertKlausewitz@reddit
opening Outlook
daisydomergue81@reddit
Were you in the artemis II mission
KlausBertKlausewitz@reddit
Unfortunately not. ;)
Operation_Neither@reddit
There are not tickets if I don’t open outlook.
Valkeyere@reddit
Outlook is your ticketing system? Condolences.
statix85@reddit
I believe that is his point
jc_denty@reddit
"I have two Microsoft Outlook's and neither is working"
Avro_Wilde@reddit
Not a disaster per se, but failed backus. Determinig why and how to resolve rhem always takes forever.
foundapairofknickers@reddit
A "quick Terraform update" for a cloud based VM
Ok-Double-7982@reddit
User who says "it's been slow for months" gets told, "Well, I was notified now that it's slow, so we will put this in our queue and get to it. There is no level of urgency since you said it's been months."
M4niac81@reddit
Pet hate of mine those tickets. It's amazing how they always have an urgent deadline for a director or something along those lines. Your lack of action in not telling me about the problem when it started happening is not going to become my emergency.
Geminii27@reddit
And then everything they tell their boss and the director is "It's been like this for MONTHS and the IT department has refused to help."
jdead121@reddit
This one. It's always a delusional person that LOVES to work too.
T_Thriller_T@reddit
Not perfectly fitting, as this is rather "a few hours across 2-3 days" and somehow ALWAYS ends up taking 2 weeks, but:
Getting a machine. Not a physical one, but a VM, configured such that it can communicate where it is supposed to communicate.
Multiple companies, it has been a hassle EVERY TIME. templates, no templates, ticketing, direct contact. It always takes forever
Symazx@reddit
absolutely anything done on a friday afternoon
neoKushan@reddit
Rebooting literally anything.
piedpipernyc@reddit
That one ticket on Friday, 1 hour before closing.
GremlinNZ@reddit
That client that likes calling at 1655 on a Friday.
Odd... They mostly don't get through?
Aevum1@reddit
we replaced endpoint provider from Symantec to another provider due to company policy.
Some genious in HQ sent a GPO to uninstall Symantec and install the new one, for some reason on one third of the PC´s the Symantec uninstall if done incorrectly would basically erase de boot partition,
at first we reinstalled but then i learned to restore the boot partition with a windows recovery disk. something that should have been simple and done automatically crippled a large number of PC´s
silent3@reddit
The annual UPS WorldShip update complete with rate changes and SQL database upgrade. It’s better than it was ten years ago but it’s still a pain in the ass.
AssociationHot166@reddit
we recently updated our UPS world ship instance from 2019 -> current. It took down our shipping dept for 4 hours..
zaphod777@reddit
Exchange updates always take longer to install than I remember, even when nothing goes wrong. I also go over everything with a fine comb before hand to see if there's anything that might potentially cause an issue.
ukulele87@reddit
You ask for IT task, so maybe something not sysadmin related from the distant past.
When i started doing IT support, somehow the psycho owner of the company only asked for me when he had an issue.
He was tech-illiterate, used an ipad for everything, refused to use a computer.
If he had trouble printing, he would say: "Cant print" or "An email is missing", and you could not ask him for clarification or any further information to help troubleshoot. He would just answer: "If i knew what was happening i would fix it myself".
Luckily his secretary was a modern day martyr who lived on the edge of a heart attack with her body pumped full of cortisol 24/7, and she would usually have some insight on what the actual issue was.
Looking back it was not that bad, but being my first job on IT, i was way too green and that whole place used the stress/fear of the employees to work (and to my surprise did so relatively well).
Ive had my fair share of bad/stressful situations since then, but i never suffered like i did there.
Unhappy_Clue701@reddit
That ‘just a small task’ thing applies to anything you start at 3pm on a Friday.
ddmf@reddit
Changing the password for that supposed one service that can't use a group managed service account, finding out it's not just one service that uses it.
Yes Wendy, I'll look into why you didn't get your early morning reports.
natflingdull@reddit
Name changes
MundaneUpVote@reddit
Peforning the maintenance the Gold Image on Fridays
kvorythix@reddit
every damn printer issue turns into a scavenger hunt because one random service account or GPO is doing the dumbest thing possible
-King-K-Rool-@reddit
Anything that involves my network technician
ErrorID10T@reddit
Anytime my client (and I'm referring to a specific client here) decides to "help."
delioroman@reddit
Windows updates
Only_Helicopter_8127@reddit
DNS changes.
Touch one record and suddenly half your services can't resolve anything, users can't hit internal sites, and you're frantically checking propagation while everyone asks why email stopped working
Ill-Mail-1210@reddit
Reboot of a particular rds server. Roughly 50 users and client insisted on admin access so they can take care of the small tasks. Last time they installed o365 business standard as they thought it be a useful upgrade over their volume licensed copy of office. Jfc. Almost every reboot (perhaps 2-3 times per year) results in disaster
NeatAdhesiveness9340@reddit
bios update
isotycin@reddit
Anything with printers
Bogus1989@reddit
Lmao, anyone in this field….may not respect it at first and think its a bunch of superstitious BS….and be my guest, hold out as long as you like. It doesnt matter how long, its inevitable. It will only be more painful.
Eventually, everyone learns to respect and understand “Read-Only Fridays”.
XanII@reddit
SMTP in prod.
Cyberg8@reddit
Smtp2go is what we use and it’s so much easier to maintain