Your worst fuk ups | TheaterFire

[-]

PassmoreR77@reddit

I was fresh and green, learning iscsi, and who knew that "initialize" meant format disk?! Who created that term! In 5 seconds destroyed 3 months of work for a dept. Backups? Yes, but the iscsi/nas was a new tech we were testing so that volume wasn't included... I was fired 3 days later. Learning moment lol

[-]

homelaberator@reddit

The first big lesson you should learn is "how the fuck am I going to put all this back together once I've taken it apart?"

Experience will teach you to expect things to go wrong but also that people will expect you to fix them. So inevitably at some point you start thinking "when this fucks up, how am I getting it back to how it was?"

After that, everything is fine.

[-]

Tymanthius@reddit

Why the fuck were you fired for wiping a testing environment?

Whoever made it production w/o back ups should have been fired.

[-]

PassmoreR77@reddit

yeahh.. it was interesting experience. But that taught me to make sure I know what is being backed up and not, and before doing anything to anything make sure there's a current delta.

[-]

Icolan@reddit

Too bad your employer decided they didn't need someone who had learned that lesson at their expense.

[-]

GrayRoberts@reddit

I signed a contract with Broadcom.

[-]

sudonem@reddit

F

[-]

keivmoc@reddit

F

[-]

noitalever@reddit

F

[-]

buzzlit@reddit

F

[-]

First-Structure-2407@reddit

F

[-]

Noble_Efficiency13@reddit

F

[-]

tenormore@reddit

Helping a colleague with some simple tasks at a client. Putting an APC UPS on the network for management. Easy change right?

Well, those UPS's had a flaw where when you plug in the network cable, they shut off. Full network outage in the middle of the day. Not the biggest disaster but still embarassing.

[-]

imabastardmann@reddit

I have also done this

[-]

jaysea619@reddit

I’ve only seen this happen when you plug into the serial port with a Cisco console cable or a network cable.

[-]

z0d1aq@reddit

The same happens with APC UPS when you connect not genuine serial cable to DB9 connector on it. Serial pinout is different, that's why. and.. it sucks..

[-]

simask234@reddit

How else are they going to overcharge for an (otherwise perfectly normal) serial cable?

[-]

WokeHammer40Genders@reddit

No flaw, completely intentional

[-]

SquirrelOfDestiny@reddit

My worst 'fuk up' was going into IT instead of becoming a carpenter or finishing my civil engineering degree something. I just wanna build something that lasts.

[-]

First-Structure-2407@reddit

Yeah, if I had my time again I’d be laying bricks and building houses

[-]

CptBronzeBalls@reddit

That was one of my big problems with IT. Everything you build will be irrelevant in 3-5 years.

[-]

dk_DB@reddit

Ah - there is the problem.

If you want things to last, you cleared need to implement more of those temporary solutions....

[-]

Brilliant-Nose5345@reddit

you think other careers dont have their own flavor complaints? grass isnt always greener

[-]

afinita@reddit

I have a coworker that always talks about “man, construction would be so much better.”

I’m over here thinking that working in either 100 or -10 weather instead of a climate controlled building is the first hurdle. Next is not having a bad back by middle age.

[-]

halxp01@reddit

The people that built server 2003 made it last, because it’s still out there.

[-]

GroteGlon@reddit

Well, you can still switch.

[-]

SquirrelOfDestiny@reddit

Every few years, I think about switching, either sideways within the industry, or completely out of it. But I make too much money. First world problems, I guess.

It's partly why I've decided that 2025 will be my 'year of health and hedonism', where I spend my time outside of work trying to live a more fulfilling life. I'll reflect on how it went in December.

[-]

GroteGlon@reddit

If you make too much money, maybe you have a little extra for some basic woodworking tools?

It's nice to have stuff you made yourself in your house, and it's nice to get away from computers sometimes.

[-]

georgiomoorlord@reddit

Yeah get yourself a shed, put tools in it, and when you've finished work go out and make summat

[-]

chefnee@reddit

This is a new year. Hopefully it will become so.

[-]

ConfectionCommon3518@reddit

Having to patch an old mainframe to support newer drives but we was one of the only sites that still had some ancient drives and the patch removed them from being supported and suddenly things didn't work and a few hours of overtime fixed it with help from the supplier but at least things got working quickly.

For true 🦆 ups it normally involves letting a sparky into the room and hearing them say oops when working live and the lights flicker and the phone suddenly melts.

[-]

7enty5ive@reddit

Restoring a veeam backup directly on the server..

[-]

eddiehead01@reddit

Deleted 1 million plus invoices from our ERP because I forgot to comment a line of my SQL script

[-]

Grandpaw99@reddit

That purple cable.

[-]

The-IT_MD@reddit

As a junior techie many many years ago, I pulled a 2u Dell from a rack, the cable management arm snagged and pulled both power cables from it.

Is was running WS2003 and Exchange 2003 for a 100 person business.

[-]

BenDestiny@reddit

My worst was that I accepted a job with a much lower position, with the promise of quick career progression as the department was brand new. 3 years down the line I am still waiting for even just reaching my original title, never mind that I was doing projects that I have never even done before.

[-]

MrCertainly@reddit

Not unionizing.

Not discussing my pay with others.

Believing when I was told by my employer that those two things were illegal and could get me fired.

(Both are federally protected, and if they even so much as gently discourage you, they can be in for a world of pure rectal pain when the government fines them & they get lawsuits out the ass.)

[-]

Affectionate-Grab510@reddit

Tried to crack a server administrator password from the previous admin do had died suddenly with a pc unlocker disk. Broke AD completely. Server useless.

[-]

F7e7t7c0h@reddit

We had a Software Distribution and sometimes build packages to fool around with other guys from the IT team.

One day I had to do a “emergency patch” but when selecting packages I tickt one box to much. Long story short, I pushed a script to ~2.5K devices that registered a job that opens the browser in full screen displaying a GIF of a Cat googling “You Suck!”, every 60 seconds…..

It was a grate day for the Helpdesk….

[-]

Haunting-Prior-NaN@reddit

While cleaning up a DFS file system I deleted a production replica. When I noticed my mistake I tried to restore from backup only to notice the backup were being done out of a replica that had not been replicated for 4 months.

I spend a long two days recovering stuff with recuva, reviewing our backup strategy, apologizing to production and swearing on my carelessness.

[-]

severs_down@reddit

I restarted a server everyone was working on.

[-]

Training-Manager-352@reddit

gif

[-]

LForbesIam@reddit

This is how we identify if people still use the server or not. 🤪

[-]

keivmoc@reddit

Good ole scream test.

[-]

EnPa55ant@reddit (OP)

Bro this made me laugh. I imagine someone yelling from the other side of the office

[-]

LForbesIam@reddit

Well no one can beat Crowdstrike ever so much so a common phrase is “Well at least you didn’t pull a Crowd Strike”

We have a lot of triple checks and multiple backups process from years of experience of how things can screw up with a single click.

Mine was I copied sysvol to multiple backup locations not realizing in 2000 how it was different from NT and it actually created junction links NOT actual copies. Then 6 months later I said don’t need these old backups and deleted the contents not the link and wiped out my entire sysvol.

Luckily I was able to spin up a new test domain and copy the default contents and in those days I printed out all my policies so I had copies of all my settings but it took me 24 hours to recover it all.

[-]

ARPG_Hobby32@reddit

Setting up 2 hyper-v hosts with shared storage, not knowing you need a CSV for two hosts to access the same storage and not corrupt everything. What a way to learn.

[-]

dunnage1@reddit

Making the decision for management to upgrade a problematic router that was costing valuable business each day.

[-]

Xesyliad@reddit

Many many years ago (we’re talking 6.4gb HDD years, late 90s early 00’s) on a customers primary raid 5 array which was configured without a hot spare… I pulled the wrong drive after a failure. Yep went about as well as you could think.

[-]

DeathRabbit679@reddit

Meant to type mv /dir1 /dir2, accidentally typed mv / dir1 /dir2 . That was fun 4 hr disaster recovery fire drill.

[-]

joshthetechie07@reddit

I restarted a terminal server where all our client's users were actively working.

[-]

No_Adhesiveness_3550@reddit

My second week, I keyed in the wrong DNS server for the PDC. Could not log back in to fix it. My only regret is I didn’t cry for help sooner.

[-]

adamixa1@reddit

i have few but i will start that happen recently

We have a 7 year old server in our server room laying and i proposed to use it as a proxmox host. Approved and without checking the server, i formatted it and ran pve.

Not so long after, we have an audit and the auditor asked us to provide tickets from 2020 until 2023 ( the current year ). Ok it's easy, but i remember i deployed a new ticketing system on the new server so the db is still fresh. The old ticketing system we kept in one server, the one that converted to pve.

My heart stopped that day. I tried looking if i accidentally imaged the server but no. So for the audit, I did not get fired yet but need to check from email notifications one by one

[-]

dfoolio@reddit

I once robocopied the wrong direction. Needless to say, thank God for backups.

[-]

noitalever@reddit

I deleted all the vlans on a core switch because i was on the wrong page and hit the wrong button. No confirmation, nothing just a spinning circle showing me i was no longer connected to the management interface and a feeling of dread as my phone on my desk went dead and my computer showed no internet… called a friend and he told me to go pull the power and hope it hadn’t committed… it had not. So the boss thought i unplugged the wrong thing and I learned to be very careful with the ui. I prefer command line.

[-]

Bane8080@reddit

Not joining the Navy instead of going into IT when I was young enough.

[-]

Gamerguurl420@reddit

The grass is always greener… spend a few months on a boat with no personal space hearing dudes whacking their shit and you would’ve wished you had gone the IT route

[-]

chefnee@reddit

Wow. I had a Navy buddy. He worked on a sub, and can’t stopped talking about the experience. I understand now.

[-]

occasional_cynic@reddit

I know if I stay another year and continue to do great work I will show my value and they will promote me then

I made this mistake in THREE separate jobs. I desperately needed a career mentor when I was younger.

[-]

Odd_Struggle_874@reddit

Well can you be my mentor now ? Desperately looking for one :D

[-]

TheTipsyTurkeys@reddit

So, what's the secret? Job change?

[-]

themanonthemooo@reddit

Job direction.

[-]

keivmoc@reddit

Truly the biggest mistake we've all made.

[-]

dancingmadkoschei@reddit

Man, all these make my basic ignorance feel mundane by comparison. Learning today for the first time that *-LocalUser commands affect AD when run on a DC? Accidentally sending out temp passwords before the mailboxes they're for are created? Nearly starting a SQL update on a live server during business hours?

Yes I'm painfully new to the large-scale side of the industry.

[-]

Gawdsed@reddit

We had storage space issues on our primera arrays, i was fairly new to storage admin. I understood that our vmware environment should be thick eager zero to let the array dedup. My storage admin mentor told me to start migrating the storage of vms from any other type to thick eager zero.

Something went horribly wrong and didn't dedup properly. Array filled up and our 800+vms stopped having storage. We had to delete useless data from our storage to get it going again.

We now have greenlake managed services because our upper management don't trust that we know what we are doing and are spending 16 mill on managed services over 4 years instead of 2 mill per 4 years.

They also refused to provide me with any training and we have no test environment or time to learn.

I accepted that this was ultimately my mistake but they understood risk... In the end i learned to go much slower with storage arrays lol

[-]

amcco1@reddit

I had a Windows server that had RAID1 boot array with 2 drives. So 2 drives mirrored. One of the drives died. So I went to replace that drive with a new one and accidently pulled out the working drive, instead of the dead one. Thus killing the drive and bricking Windows.

Luckily, this was just a backup server. It was a 2nd location backup for HyperV replication. So it actually didn't hurt anything, but I had to rebuild the server, reinstall Windows, setup replication again.

Wasn't really that bad,, but that's the worst I've done.

[-]

dmuppet@reddit

Happens at least once or twice a year. Remoted into a hypervisor bc a server isn't accessible. Go to restart server and accidentally restart the host.

We turn those into scheduled failover tests.

[-]

Lonestranger757@reddit

Inbetween Phase trying to Learn SCCM and get away from WDS... I deleted the task sequence on SCCM...little did I\we know that it was directly tied to Imaging all Desktops and Laptops from the WDS serverfor a 1500 computer Org.....my admin rights where quickly revoked from that box.....

[-]

crashtesterzoe@reddit

Early in my career I had exchange 2003 and tried upgrading it directly to 2012 as it hasn’t been updated in years before I got there. Well probably can guess how well that went. A week of exchange being down. What a stressful first month of work 😂

[-]

SystemGardener@reddit

I brought an over 1000 office phone network completely to its knees and down for about 6 hours.

They had a shockingly small circuit for that amount of phones. They also had firmware updates due on all the phones.

So ya… you can probably guess where this is going. I pushed the update, to all of them, at once. I guess standard procedure for this site was to do them in much smaller phases due to the circuit size.

I immediate escalated after seeing everything go down. Luckily it was after hours and the higher tech suggested we just pray that it sorts itself out before morning and that this isn’t the first time this has happened. I still have my boss the heads up just in case. Luckily everything sorted itself up by morning an all the phones got the update.

[-]

FunkyAssMurphy@reddit

First couple years in the industry working at an MSP. I was in the server room livening up some jacks. The rack management was those heavy metal sheets that go on the left and right side of the rack.

I was fighting with it to get it back on properly, but it was old and a little bent. I sort of had it, looked away for a second to grab a mallet to tap it into place and it slipped off on its own and fell straight down.

It landed directly on a power cord that was like the 2nd link in a daisy chain of 4-5 power strips (not our doing I promise).

Well it cut power to 80% of the rack and plywood next to it. Fried a few modules on their nortel phone system and one of their network switches.

Luckily that’s what insurance is for, but boss wasn’t happy

[-]

fang0654@reddit

Long ago I had a small IT consultancy in NY. Had a client that wanted me to expand the storage of their SBS, which hosted their DC, their email, etc.

When I went onsite late Friday I asked about backups, and they showed me their USB backup. So I go in, use ntfsclone (or something like that) to image the drive over to my own USB, then blow away the raid, add in new drives and rebuild raid. Image everything back, reboot, and things start failing. Long story short, the drive was super fragmented, and the tool didn't copy all of the files, just the first few blocks. Then I find out that the backup is two years old.

I spent the entire weekend rebuilding from scratch, pulling email from local profiles, etc. It was a nightmare.

[-]

TheTipsyTurkeys@reddit

Became familiar with Kaseya 365

[-]

8923ns671@reddit

Forgot to apply the new load balancing I just setup leaving the customers solution in a non-functional state for an hour after the change window. Oops.

[-]

exterminuss@reddit

Deleted the no 2FA group at time where the ViPs where not yet migrated to 2FA

[-]

Catfo0od@reddit

I was the one that sold Ricky the smack! How was I supposed to know it was laced, huh?!?

[-]

klassenlager@reddit

I forgot add on cisco switch… switchport trunk allowed vlan x

Took their whole network down for an hour

[-]

Naclox@reddit

Back in the late 00s I was in charge of the computer labs at a public university. We had just gotten Symmantec Ghost to do the imaging of all of the computers and were using it. Not knowing nearly as much about networking back then we were testing the various options of unicast, multicast, and direct broadcast.

We found direct broadcast seemed to be the fastest method so we started imaging a bunch of computers in one of the labs one evening. At the time the university didn't have firewalls between subnets and a lot of places were on 100mb and some places were still on 10mb switches. We had brand new gigabit switches connected to our imaging servers and a lot of the desktops we were imaging. We ended up taking down the entire university network with a huge traffic storm. Fortunately phones were still analog so when the network engineer figured out what was going on he was able to call me and get it killed.

You would think the story ends there, but it gets more interesting even though I wasn't at fault this time. After some brief training on how the protocols worked we were told we should be using multicast. The next night we attempted to image all of those computers again using multicast. I again get a call from the network engineer telling me we were sending broadcast packets across the entire network and he was pissed that we did it wrong again. I double-checked and we were indeed using multicast, NOT broadcast. We shut that down and the next day we (me and the network engineer) started investigating. It turns out our brand new gigabit switches (which were also used in the network core) had a firmware bug that made any multicast packet a broadcast packet instead.

It took the hardware vendor weeks to find and patch that bug. I don't think we bought any more of their equipment after that incident.

[-]

Jeff-J777@reddit

I was working at an MSP on a clients network.

I was working on an Exchange issue between on-prem and 365. There was an issue with our barracuda and we had to re-route in-bound emails to 365 and allow some to reach the remaining on-prem mailboxes. Well I was adjusting some firewall rules trying to get data to flow. In testing I opened the policy wide open to the internet on all ports.... I KNOW I KNOW I was also doing this on a late Friday night with a pounding headache. Well someone else got a different work around going so I bailed and pass out. But I also forget to clean up the testing policy I made on the firewall.

Two weeks go past and the dev team see a bunch of sa login attempts on the core SQL server. This is a hefty SQL server with almost a TB of ram and 20TB in databases. In a frantic run the IT director tell me take them off the internet. I kill the firewalls. They engage with their cyber insurance, analysis is done, and they said well there is just a wide open policy to the SQL server on the firewall. I say impossible. Well I was wrong here I mis typed the inbound NAT policy and set the internal IP address to the core SQL server and not Exchange. Well I figure out why the Exchange work I was doing week ago was not working.

But I thought well I just lost us a huge client, and possibly my job. But I did not try to hid what I did I owned up to it I walked into the IT directors office and said "my bad". But the other saving grace was the monitoring we had in place as part of the MSP contract was working for week and being completely ignored by the two sys admins. We configured the monitoring but they wanted their in-house sys admins to watch it. Well they received of 20k of emails alerting them of the failed sa login attempts.

I lived there for days doing everything I could to fix the situation I had caused.

In the end nothing bad happened. The SQL server was never compromised, no data was access. HUGE RELEIF!!! They ended up giving us a lot of security projects to fix things up. But those few days I was freaking out.

[-]

shanxtification@reddit

I've had a few lol. Most memorable one was doing a quick battery swap on an APC UPS 1500. Couldn't quite reach the connector lead to plug in the new battery, so I whipped out my trusty pliers and accidentally completed the circuit internally, frying the UPS and brining their ancient servers down. What was supposed to be a quick 15 minute job turned into 3 hours of me getting their servers back up and running properly.

[-]

ISeeDeadPackets@reddit

Restored a several month old very important DB server over top of the production one instead of to the sandbox. Thankfully someone had anticipated his own capacity for stupidity and had a very recent storage snap handy. Now if I'm doing anything with that kind of potential I shut my door, sign out of Email/Teams/etc.. and unplug my phone.

[-]

martinmt_dk@reddit

Not entirely only my fuk up - but a combination of fuk ups caused a major one.

Back in the old days, back when Hyper-V was new and fresh, and virtualisation was weird and new. Back when most servers consisted of using a dedicated server with 10 different applications on them to save on hardware. Back when backups was just something your needed in case of a raid dying and before ransomware etc. was ever a thing.

Anyway, long backstory. We had a fileserver on Hyper-V (our first), I needed to expand one of the disks which i did. Unfortunately for me, there was a 6 month old snapshot on this disk. Hyper-V had that great feature (not sure if that's still the case), where it broke the link between the vhd and snapshot file. So after the disk was expanded all data that was changed in those 6 month vanished. (my Fuk up)

Anyway, no panic, we have backup. We used TSM, which basically stored all the data at an external vendor. So i tried to restore it, only for the job to keep failing. Turned out the vendor didn't really focus that much on the restore part of data, and it ended up taking 24 hours for them to get a restore started. However, this restore only worked on 1 "stream" instead of multiply, and it could only be restored locally.

So basically, the restore process ment that a server in the vendors datacenter was attached to a disk, and then TSM would restore the files in serial - one file at a time - After the restore was complete they drove the disk to our location, after which we could robocopy the files back on the server.

If i recall correctly, it took about a week for the server to be somewhat back online again.

[-]

retbills@reddit

Mimecast has a less than ideal UI so my goal was to block attachments from a specific sender however I managed to block outbound attachments for four hours which you can imagine the chaos. However, silver lining was that it occured on a Friday when all shipyards piss off early for the weekend so the impact was not as large as you'd expect.

[-]

InfamousStrategy9539@reddit

God, I despise Mimecast’s UI

[-]

malikto44@reddit

Three things:

Not focusing on building a wrecking crew that would look out for each other and the whole crew would jump companies as a gestalt. Social stuff is far more important, and word of mouth gets and keeps you employed.
Getting my degree. I should have, once out of high school, found a job at Dell or a PC company and worked my way up, perhaps grinding out a unique niche. That, or going into the Navy and getting a TS/SCI clearance. Even a focus on certs would have been more useful. I do have a degree, but paid severely due to opportunity costs.
Not bail fast enough. If your company gets bought out, GTFO. If you are onboarding contractors or an offsite place, run. It gets easy to just coast at a place until the bitter end... don't do this.

[-]

New_Worldliness7782@reddit

Coded a solution to anonymize data in compliance with GDPR regulations. Made a mistake in my SQL query, which resulted in all master data for dealers who, at any point, had a customer that needed anonymization being anonymized. About 10 minutes after running the code, I could hear the advisors on the same floor as me answering their phones, saying things like, "What do you mean your data is anonymized, and you can't submit cases to us?" I started sweating and realized I had made a mistake. It took 4 very stressful hours to restore the master data.

[-]

iamLisppy@reddit

Staying at my last job for too long.

[-]

stratospaly@reddit

Early in my career I was told by a Senior to "reseat these three drives on the SAN". I questioned this and made them repeat it three times because I knew just enough through certs to know that would not work out. I was to "shut up and do as your told". So I did it. The entire backup array borked and I was blamed. After 5 full minutes of my bosses boss yelling at me when I was asked what I had to say for myself, I explained I was on the phone with X senior who told me exactly what to do. I was then told that I should have read between the lines and reseated the drives one at a time, and not done as I was told if I knew better. "The Nazis were just doing as they were told!" was the example given to me. The conversation then turned into how we would recover from the mistake. The Senior spent a full week trying to recover the data that were just the on-prem backups (we had offsite intact) rather than listening to my suggestion of zeroing out the array and starting backups over starting today... Which eventually did, claiming credit for the idea, after we lost 8 full days of backups at this point because we could not send offsite, what never backed up in the first place.

From that point on at that job everything I did was inspected under a microscope as If I were the largest fuckup imaginable, where the senior walked on water and could do no wrong.

[-]

womamayo@reddit

My worst one was I updated firmware on our stacking network switches and thought it was a beautiful Saturday but guess what? all switches became bricks. holy shxt I was dead inside my brain. Ended up I found there has few old 10/100M switches at the corner of the server room and I spent whole weekend to create network environment, checking everything can run. btw, I am the only IT in the company..

[-]

Creative_Onion_1440@reddit

My worst fuckup was when I was doing Avaya switches and the AV team wanted one in their office with all the building VLANs on it so they could test equipment without needing to go to the different buildings. Anyways, I put EVERY building VLAN on one trunk to a 24 port ERS4800 access switch and it took the entire campus down.