What's your best troubleshooting story? And your worst?
Posted by legendov@reddit | sysadmin | View on Reddit | 46 comments
Best Troubleshooting Story:
Back in 2005, I was working at a Dell call center. A guy called in complaining that his monitor would randomly display weird colors and start "shaking." Screen sharing didn’t show any issues, so I figured it had to be a physical problem.
We went through all the usual steps: checked the connections, reseated cables, even power cycled everything. Still no change. The monitor kept acting up with those strange colors and movements.
As we talked, a thought hit me out of nowhere. I asked, “What’s around your monitor on your desk?”
He started listing things: papers, a stapler, and then casually mentioned his desk fan. Bingo. I asked him to move the fan to another part of the room. The issue immediately stopped.
Turns out, the magnet from the oscillating fan was interfering with the old CRT monitor. The client was floored, and I felt like a troubleshooting wizard.
Worst Troubleshooting Story:
Fast forward to 2012. I’d been out of IT for a couple of years, tried a different career path that didn’t work out, and landed a gig as a junior sysadmin at a small MSP. It was my first call out, and I was eager to make a good impression.
The issue? A PC was “slow” and acting up. I ran every test I could think of: stress tests, diagnostics, you name it. Nothing was wrong. After an hour, I told the user, “I can’t find anything, but maybe we should wipe it and start fresh.” They agreed, saying they’d back up their files, and we’d reformat in a couple of days.
I left and headed back to the office—a 30-minute drive. When I walked in, everyone in the office was staring at me like I’d done something terrible. Confused, I asked, “What happened?”
My manager pulled me aside and explained: ten minutes after I left the client site, the PC caught on fire. The entire office had to be evacuated.
Thankfully, the client laughed it off, and I wasn’t in trouble. But let’s just say I’ve never been able to top that for sheer chaos during troubleshooting.
Prismology@reddit
Worst troubleshooting story:
Installing a new second monitor for someone. Fully plug it in. Turn it on. It isn’t turning on. Boss comes over from a different building in 100 degree weather. The second he stepped in the room the monitor magically powered on.
“Why’d you make me walk all the way over here, you can’t even hook up a monitor by yourself” right infront of one of my professors
Probably_a_Shitpost@reddit
Got hacked. Encrypted everything and all admin accounts pw changed. On the call with the 3rd party DR team they showed us logs that kept showing the same 16 digit garbage. So I copied it and pasted it into my domain account and BAM! WE GOT ACCESS BACK
_mux_@reddit
Fuckin dell. I like the optiplex 5000 series, esp the 5090s. Then they went to 7000 series. We order micros. Dumb things have 3 DP ports on them. 1 less usb port, ugh, but I digress.
Most users have two screens. Sometimes, no matter what ports I used, 1 monitor wouldnt get a signal. Losing my mind. A lot of users have to use dvi to DP adapters.
Finally, figured out only the brand of adapter matters to these little shits. I forget ehat was the shit brand adapter though.
I was a tech back then.
modernknight87@reddit
While I was in Afghanistan, I was a volunteer web admin so I could help personnel create accounts and brief people on Cybersec issues (my actual job was Air Transportation). One day we needed to apply some updates to Server 2008. Up to this point I had never seen the box - hell, no one had. So I drive all around Kandahar Air Field and find the NOC. THEY have no clue where my server was. As I started looking around one of the network guys says “Well, we have this small box in the middle of our server room and no one knows what it is..”
Sure enough, a year and a half it had been sitting there and not a single person knew who it belonged to.
I felt like a champ after that, given no one else had been able to find it before 😂
Robeleader@reddit
Classic from bash.org (which is apparently dead?)
Otherwise-Ad-8111@reddit
I remember working in a call center and a sweet old lady called having issues with here TV. It was July 8th, 2011. I remember it vividly because it was when the last space shuttle launched, and the little old lady asked if the reason her tv wasn't working was because the space shuttle was parked in front of a satellite.
--
Different job, but had another IT person (making way more than me) ask me how close their RSA Token needed to be to their computer for it to work.....
---
the absolute worst was we had an admin assistant that basically worked by herself in one of our furthest regional offices (nearest tech would have to drive 7 hours to get to her). She decided to re-arrange her office, and got her phone and computer plugged back up incorrectly. We had detailed notes of this lady's office because she was difficult to deal with. Specific colors of wires that were clearly labeled so we could walk her through it. She did NOT want to be helped over the phone, she demanded we send a tech. I escalated internal to my management.....and we rolled a truck 7 hrs away to swap two cables :)
mfinnigan@reddit
I once tried drunkenly unlocking my car with my RSA keyfob.
(I was trying to get my overnight bag so I could brush my teeth before going to sleep at a friend's house, I was not trying to drive drunk)
WyoGeek@reddit
I have a couple that stand out.
I got a call from a department head that their computers were all randomly rebooting. I went over to verify the problem and while checking things out, sure enough 5 PCs's all crashed at the same time. I knew it had to be a power issue so I started tracing everything back. What I found still makes me shake my head. Everything was plugged into one wall outlet and there were 6 power strips daisy chained together. At the very end of this electrical shitshow was a small refrigerator. Every time the refrigerator compressor motor kicked on it would cause a brownout that crashed everything. The crazy part was there were actually wall outlets nearby each PC so once everything was plugged in properly, there was never a problem again.
This reaches way back to XT PC's with 5 1/4" floppy drives. A user called me reporting that her PC was eating floppy disks. I figured there was a hardware problem with her floppy drive and it was physically hurting the disks. I grabbed a replacement drive and my tools and headed to her desk. I asked to see one of the disks so I could verify the problem to which she said no, it's actually eating the disks. I put the disks in and they disappear. I grabbed a disk and had her show me how she was inserting the disks. She proceeded to stick the disk into the tiny space between the disk drive and the blank plate above it. I opened the case to find at least 10 floppies inside just laying on the motherboard. I showed her how to properly insert and remove a diskette from the drive and needless to say she was more than a little embarrassed. She was such a nice lady and I felt bad for her.
weauxbreaux@reddit
When I started out in IT, I worked at an AD Agency. The 'Creative' teams got Macs, because, well Macs are better for design, duh!
One day, I got a frantic call from a new employee.
"Listen, something is wrong. My mouse is broken! What is going on here? When I move it up, the cursor goes down! When I move it down, it goes up! When I move it left, it goes right! This is seriously broken!"
"Ok...Ok... hold on a second. Look at your mouse. Do you see the little Apple logo? Which way is it facing? Is it on the Top or the Bottom of your mouse?" (this was a magic mouse)
"NO NO NO.... OH. OK yeah that did it thank you!"
I'm not sure if this is my best or my worst
Zenkin@reddit
"Whenever I type in Word, things get crazy and my letters just show up in different places in the document. What's happening?"
"Are you typing on your laptop?"
"Well, yeah, but wh-"
"You're hitting the touchpad with your palm, which is 'clicking' wherever your cursor happens to be at the moment. We can get you an external keyboard, or maybe reduce the sensitivity on your touchpad."
"Hah, well, I appreciate you trying, but that's not what's happening here. I'm not touching the touchpad."
"Soooooooooooooo..... I was touching the touchpad when I was typing. My bad."
ewok66@reddit
We used a fan to trick our boss into thinking our (massive) CRTs were bad and needed to be replaced with flat screens. Couldn’t believe it worked…
Flatline1775@reddit
Best was back in 2012. My home computer was having an issue. I don’t even remember what the issue was, but the fix ended up being clearing a single registery entry.
Fast forward about a week. First day of my first civilian IT job after 8 years in the Marine Corps. I’m sitting in my new mangers office and one of the other techs pops her head in and says they’re still having an issue with several systems. She starts describing the issue and by god it’s the same issue I had on my home PC a week prior. So I tell them they should try clearing out this registry entry. (I didn’t remember the exact path, but was able to walk the new manger through on her system.) They both look at me like I’m insane, but the tech leaves.
She comes back like five minutes later and says it fixed the problem. Some real Slumdog Millionaire energy to that one, but it immediately put me on good footing with people.
AmiDeplorabilis@reddit
To them, it appeared that you pulled a rabbit out of your hat... all too frequently, I have Bullwinkle's hat!
rheckber@reddit
Sysadmin at a university but years ago we were on overflow for help desk calls. Got a call from a female student telling me her laptop wasn't working and could we take a look at it. I asked her the symptoms and if she had spilled anything on it or . . .
Her reply was a classic - I was using it, I dropped it, there was a spider involved . . .
AmiDeplorabilis@reddit
That's almost forgiveable...
SilkBC_12345@reddit
Not my story, but my all-time favorite is the came of the 5-mile e-mail:
https://www.ibiblio.org/harris/500milemail.html
LOLBaltSS@reddit
That and Bedlam DL3 are my favorites as an email admin.
https://techcommunity.microsoft.com/blog/exchange/me-too/610643
Another one I liked was the case where iPhones and other Apple devices got disabled at a hospital because of a helium leak. https://www.ifixit.com/News/11986/iphones-are-allergic-to-helium
Draptor@reddit
Some actual testing of the helium issue from a darn good (if sparsely posted on) youtube channel.
JimTheJerseyGuy@reddit
Truly a classic. Sendmail.cf take me back. And not to a good place.
driodsworld@reddit
One of the classics
legendov@reddit (OP)
This is unbelievable lol amazing
PresNixon@reddit
In the early 2000s, someone called and asked me if they could have a third monitor because they were running out of hard drive space.
Their desktop was FULL of icons everyone. I showed them how to make folders on their desktop and move things into there, and eventually how to put those folders elsewhere.
RuggedTracker@reddit
We connect to our clients mailboxes to provide some value for them (of dubious worth, but I'm not the one paying the bills so w/e). One day we started getting a lot of error messages from one client, and quickly determined that their mailbox was full and Graph was refusing any actions.
We forward that info to them and turn off the solution, thinking we'll get a green light to start processing again soon. The day ends and I go home with my laptop to be ready to re-enable the logic app.
The next day comes and goes, still all quiet from their side. We send the information again, making sure to specify that this is production critical for their case handlers just in case their IT department was siloed away. Nothing but a "we are working on it" comes back. Working on what I don't know.
Then it's weekend and I am in disbelief that a multinational company with dozens, if not hundreds, of IT personnel are incapable of archiving or even deleting emails from a mailbox.
On monday I find myself in a meeting with their PM (regarding their integration with us), our usual contact point, 1 IT guy from their company, 2 IT guys from their parent company, and 2 consultants. All of us gathering around to discuss how to delete emails from a mailbox
Half an hour later I leave the meeting while people are patting themselves on the back for a productive planning meeting and a plan for archiving the "Deleted Items" folder
If it weren't for the fact that our usual contact person was freaking out over almost a week long backlog on claims and their customers being furious, I would've thought this entire ordeal was a joke played on me.
(We turned on the solution again later that day)
(funnily enough, only a few months later, the same error happened for another client, and I once again sat in a meeting to discuss how to delete emails. Luckily in this case the meeting started only minutes after the error was discovered and we deleted the emails during the call)
way__north@reddit
A user called me, with his PC beeping incessantly in the background, stating "help be get rid of this crazy beeping before I lose my mind!"
I had been at that location the week before, and his desk was then full of binders in differing colors as he was working on a new schedule for the fire department.
I saw the scenario for my inner eye and told him "Well, first thing you can try is to move that yellow binder to the right of your keyboard 1 inch to the right and see if it helps!"
It got quiet. And he got quiet. Before "uhh, can you please tell me where you put the camera?!?"
It turned out here was actually a need a camera to watch him - he was discharged six months later for petty thievery"
7ep3s@reddit
when i as the L2 desktop monkey had to take a wireshark trace and point out that it was dns when both the network engineer and the infosec engineer (they dns) firmly stated that it cannot be dns.(it was dns)
legendov@reddit (OP)
It's always dns
1singhnee@reddit
I worked help desk at a large, well known software company in the late 90s- I was surprised that so many computer science graduates didn’t know that fans and speakers could screw up their CRT.
Nikt_No1@reddit
I spent 35-45 minutes on the phone with 4 people who could not enable internet on their phones, no matter what I tried. At the end I just told them that probably signal in hospitals is usually bad or straight up blocks internet...
gadget850@reddit
I was senior tech support for a printer manufacturer. Blockbuster was getting our laser printers and the first store had issues with a toner out message. Sent a tech in and he called me. He was confused because he had moved the printer to a different location to work on it and the toner out message went away but cam back when he put it on the counter. Had him look around and tell me what he saw and when he got to the exit security system I had a bingo. The toner cartridge communicated with the printer via an RFID chip, and the security system swamped the signal. We were able to move the printer and resolve the issue.
JimTheJerseyGuy@reddit
Best: back in the 80s I worked in a small university computer lab that had a massive IBM line printer that was a hand me down from a larger lab. It dated from the 70s. If you’ve ever seen one of these in operation you know that a) they are loud and b) they can move around like an off balance clothes washer if they aren’t properly secured.
Well, this thing was for some reason on castering wheels, the type that have the little locks to hold the wheels in place - rather critical in this application. Normally, the printer was in the main lab area but on this day someone had moved it into a nearby, separate room while maintenance was painting and forgot to lock the wheels.
In this smallish (10’x15’) room were crammed a variety of network switches for the lab, mainframe controllers for terminals in the bursar’s and registrar’s offices nearby and the PBX for the entire building.
Someone kicked off a large print job and this beast starts shuffling itself around this tiny space. The first thing it did was to close door. Which promptly locked and no one could find the key.
It then managed to unplug the PBX. So no phones to call anyone to find the key. It then proceeded to take out all of the other equipment in the room by either knocking the electric cords from the wall or by shifting the entire equipment rack they were in.
I had had the idea to kill the print job to stop the carnage. Nope. Our terminals went through the controllers in that room too. The printer, as we later discovered, was actually connected through a controller on the floor above.
The resolution involved myself and another enterprising worker forcing open the lone ground floor window into the space to subdue the beast and power everything back up.
Worst: only job I actually had wear a suit for. NYC. First job out of college. The “problem” department was always accounting. Some of the least computer savvy people I’ve worked with, even given the early year we are talking about here.
One employee (let’s call her Darcy) was a particular source of problems. And she was very good at convincing her VP boss (my boss’s boss as it turns out) that IT was the source of so many delays in her job.
One day said VP grabs me as I’m walking back onto the floor from lunch and reads me the riot act. YOU SAID YOU FIXED DARCY’S PROBLEM! SHE STILL CANT GENERATE THAT REPORT I NEED! AND YOU WENT TO LUNCH??? THIS NEEDS TO BE FIXED NOW!!!
The problem earlier was that she’d knocked her keyboard connector loose (again) and it was working when I left. So, I sit down at her desk with her and the VP hovering behind me. Neither say a word as I start trying to type on the keyboard. Weird. It’s working…sort of. Some keys are and some aren’t. And then I’m just getting some random characters.
This department used a ton of multipart forms and the keyboards were constantly getting jammed up with paper dust and the random loose staple as they ripped and tore these things right over the keyboards. So, I did the first thing I usually did which was to flip the keyboard over and give it a good whack to jar anything loose that could be gumming things up.
And I promptly got (no exaggeration) an entire cup of coffee down my suit jacket, shirt, tie, and into my lap.
She’d spilled an entire cup of coffee on her desk and into the keyboard, wiped up everything on the desk, left the keyboard alone, and then just shrugged and went “hmm, keyboard doesn’t work again. Can’t do my job…”
I stood up looked right at the VP and said I’d be right back with a new keyboard once I’d dried myself off. To his credit, he did tear her a new one after I’d left. He also apologized to me separately and offered to pick up the dry cleaning bill.
Maxplode@reddit
Years ago when I was in MSP we had a customer who was a bit of a character to say the least, I answered his call and he was a little bit angry that he couldn't print from his tablet. We restarted the printer and the tablet then I told him to restart his router.. he said he didn't believe me that this would work and was still being a PITA.
He reboots the router then about 10-15mins later I get a call back. When I answered it it was him and the first thing he said was "How do you sleep at night?". Then he changed his tone and was laughing as it was now working.
Sadly that guy passed away some time later.
My worst one was a lady that I always managed to answer somehow and she was always travelling and for some reason she couldn't get her wireless mouse to work, this must have been like 3 different times and she was very uncooperative, also her voice was very nasally and would just f-ing grate on me >:(
333Beekeeper@reddit
1999-2000
I worked for a service company and was assigned to the Texas branch of a medical coding company. Main office was in California. The first challenge had to do with a fax server that kept rebooting. The original tech installed it with comm boards that accepted 12 POTS lines. Every time it reached a certain number of inbound calls it blue screened and rebooted.
I asked the telecom tech if the AT&T Definity system could fake a T-1 for a dummy area code and divert twelve incoming lines to phony internal numbers. He said it could. I found a Brooktrout T-1 card online.
I went to the company’s management to let them know the plan and got approval. I do need to clarify at this point that I knew nothing about fax servers or even if my proposed solution would work.
I installed the new card, connected the T-1 and rebooted. When it came back up I configured the software with the fake numbers and made a test call. It worked! I configured the software to direct each line to a specific printer for each department. Everything started flowing in from various doctor’s offices who were customers.
The branch failed about 8 months later and closed due to embezzlement by the senior administrator. Le Sigh
antons83@reddit
Best: we have a screen capture tool. One day I get the ticket saying the editor doesn't open. I connect to the user, click on the editor, the splash screen opens up and closes. No error, no nothing. I go through my regular troubleshooting steps. Reboot, uninstall, reinstall. Comb through the settings. Googling. I couldn't recreate the error on any of his coworker machines. For three days I picked at it. At the time I had just started L3. Usually in L1 and L2, we'd spend some time trying to fix it, and if it didnt work, kick it to the next level. Now I am the next level, and I'm panicking. Give this guy a new machine? All for one application failing? He'd have to spend the next day setting up all his apps, so I kept hammering at it. I had never used Process Monitor before. Again, being L1 and L2 for a long time, I never had to use other tools. We were also not allowed to download any other tools. So now I'm learning how to run Procmon. I'm scanning and scanning, and there it is. ACCESS DENIED. It was pointing to our on-prem sync location. We had just moved to cloud-sync, but staff still had access to on-prem, but just read-only. Turns out the user had downloaded a 1MB sticker pack to "jazz" up his screenshots. This sticker pack saved in that on-prem location. It needed write-access to that location. When we had switched over, the on-prem sync became read-only, the editor couldn't locate the tiny sticker pack, and it CRASHED the program. No error, no nothing. I was dumbfounded. I've also never seen this error ever again. This user was the only one that had ever downloaded a sticker pack to bedazzle his screenshots. Lesson learned.
Worst: Was trying to figure out why this users camera wasn't working. Spent about an hour banging my head against the keyboard. Talked to my coworker about it. He giggled and showed me. Lenovo T14 machines come with a tiny slider on the camera that covers the camera AND disables it. Lesson learned.
Weak_Jeweler3077@reddit
Had one of those the other day. 2000 miles away, a laptop client can't get her camera working for Teams. Started with "check to make sure the camera lens doesn't have the slide closed". She promised shee checked and it wasn't.
An hour later, it turns out it was.... She just didn't know what I meant.
Claidheamhmor@reddit
Was called to my wife's colleague's house on sunday. she said she'd reset her router and needed me to set it up again. she had pressed the button that turned wifi off...
themorah@reddit
I had a similar issue once. Had a user with one of those all in one touch screen PCs. She was complaining that the cursor would move by itself, windows would close, or minimise without her doing anything. Turns out she had a vase of fresh flowers by her computer, and when she had the window open they would move in the breeze and touch the screen and cause random things to happen.
DaithiG@reddit
Fairly annoying user was claiming her laptop was shutting down at odd intervals. Got the laptop, couldn't replicate. Gave her a replacement laptop, same issue for her. Couldn't replicate.
Finally realized she was wearing magnetic bracelets which were triggering the laptop close lid sensor or something. I knew it was something to do with the user, but didn't think of what she was wearing.
Tricky_Resolution241@reddit
A user once called me to investigate weird noises allegedly coming out of the rj45 socket in the wall of her office. She swore there are ratd in the walls and they’re coming out (like that's IT's problem to begin with but whatever). Spent there 20 minutes checking it out because she insisted and nothing. Then when I was leaving she heard it again. Well I observed. Few minutes of nothing and then I heard it. The guy from the next office behind the wall left his phone there and somebody was calling him... Hence the sounds of mouses coming out of the ethernet.
SapphireSire@reddit
Idk about best but the most annoying was a client had me build a few new PCs for them, and they demanded I come out to fix their new PC bc it was randomly shutting down.
I checked logs and it was losing power, and after them saying it's been going on for days, took me 5 minutes to look at their power structure and ancient wall plugs, dilapidated building, and then the kitchen on the other side of the wall.
Out of curiosity, I turned on the microwave and the PC crashed.
Advised them to seriously reconsider UPS systems at the very least and also a qualified electrician.
They went from literally angry at me to sheepishly quiet in less than a minute.
SciFiGuy72@reddit
Tale from the Before Times, mid 90s:
Before I was in IT, I was telesales but taking CS courses. The company I was with somehow got into Lotus Notes, despite having about a dozen employees (learned later they pirated it). Anywho, the guy over IT was coding some backend stuff for the notes environment...12 pages of obscure looking code and couldn't get it working. He comes to me since I was in CS and I ask for a printout. After reading twice, my suggestion to him was to keep the first and last pages and set a variable calculated in the first to a parameter in the last page. Of course, he's skeptical but tries it....worked like a charm. Turns out the intervening pages were his effort to work around a mathematical calculation which he'd done and dropped after it was arrived at. He wasn't around much longer and I got his spot...on probation....for the next 18 years.
BrutalGoerge@reddit
Worst, I will keep it short.
Upgraded and promoted a new domain controller server to replace an old one. For some reason the DHCP config didn't import to the new one. So I manually entered it in. Later, all of a sudden users complaining about network drops. Turn off new DC, turn back on old DC, things clear up. Confused as hell, make sure nic drivers are updated on new controller, try putting it in different switch ports. Did a lot of things that made no sense but another networking guy was spitballing which I can't remember the specifics.
Finally, I started looking deeper in my network switches, then i saw it, the ARP table showing some random tablet assigned the same IP address as my gateway.
I forgot to enter in the DHCP scope exclusion range.
Best probably was when I was new to my position, and noticed we had a CCURE 9000 site server that did not have backups. So I made my case to buy licensing for automated backups for the server, and a month after I got backups rolling on that server, its storage device failed. Just a few hours of server downtime, and zero hours of site access downtime since the controllers cache the config and can run on that for a few days.
m5online@reddit
This was in the mid 90's, and i was working retail in a big chain store similar to CVS/RitAid (They were called PayLess Drug). I was that 18yo kid who was into computers... Our store would get a download on saturday nights on its mini mainframe of all the sale items that week, and then auto print little sale price placards that we'd run around and place on the actual sale items sunday mornings.. Well, the server didnt print out the signs, and they couldnt figure out why. My brilliant plan was to just hit Cntrl+Ald+Del, because that always fixed everthing, right? So ya, I shut down the whole store. The registers, the Pharmacy database registers, everything.. My supervisor had to call corperate and they had to call in one of the sysadmins on his day off to come in and do some kind of remote fix. The store was unable to do any POS transactions for a good two hours. The ONLY reason why I wasnt fired on the spot was because It was sunday morning and the store happened to be pretty dead, and my supervisor was mortified of emberresment for letting me do such a thing that she just blamed the server for glitching.. I'm not sure if the sysadmin guy was buying it, but no one got in trouble. I was never allowed to even look toward the general direction of the server room after that....
Sir-Vantes@reddit
Mine was the receptionist who had somehow moved the Word program window about 4-6" clear of the displayed desktop.
You could alt-tab to it but out of mouse range on the Windows 3.11 desktop.
The alt-tab had the effect of flashing the border nearest the window, and using the Explorer + arrow keys was able to bring it back and get it snapped to the desktop.
Canoe-Whisperer@reddit
Best story: showing up Microsoft support on enabling modern auth/configuring conditional access for users on a RDS server. Nobody could tell me why Outlook would ask for a password upon every login. It was a long week, but a very sweet victory - nobody at MS support could figure it out. It was the RDS UPDs not catching the few appdata folders under roaming. This was years ago fyi...
Worst story: shared secrets between NPS and WiFi APs or VLANS in my junior years. Much damage was caused...
throwawayswipe@reddit
Best/Worst: Had to set up Onedrive on elderly customers' cell phones (I work remotely) and got them to install Teamviewer so I could guide them through the setup process. Best because it was a satisfactory solution and worst because it was a bit of a hacky workaround.
GoodMoGo@reddit
I spent nearly three hours with an HP printer before I called HP, and they told me not to use a surge suppressor.
Other than that, 95% of calls answered by an Indian tech center.