Ops engineer who built half our automation just gave notice. Nobody understands the system

Posted by Otherwise-Papaya-105@reddit | sysadmin | View on Reddit | 211 comments

Ok so our operations engineer just gave his notice a few days ago, and I just realized how much of our mid-size startup relies on what he built over the years. He wrote tons of automations that move data between systems, generate reports, trigger approval, and all the other QOL stuff. I mean everything still technically works and we had a good chat. (He got a better offer and I completely understand his decision, we still keep in touch from time to time, especially when I have questions.) But the thing is, nobody unedrstands how things work except him. There are some resources that he left behind, although they're pretty outdated, so now upper managemetn it scrambling asking if we can still keep things running. For those of you who have dealt with this, how do you recover when everythging is basically locked inside the automation stack of an employee who just left?

Reply to Post

Reply

211 Comments

[-]

Sudden_Office8710@reddit

Wow, why did this get removed?

Reply

[-]

ThePsychicCEO@reddit

I've had a similar situation recently and found Claude very helpful to generate retrospective documentation.

Reply

[-]

v-irtual@reddit

Go buy a copy of The Phoenix Project for everyone, and let them know you just lost Brent.

Reply

[-]

BlackFlames01@reddit

Thanks for mentioning The Phoenix Project. I looked it up and it seems like an interesting read. 🤔

Reply

[-]

PlzPuddngPlz@reddit

Should be mandatory reading in our field. It hand waves some adoption challenges but is great for realizing that every org faces the same problems & there is a path for fix them.

Reply

[-]

BlackFlames01@reddit

I see. 🤔 With such a strong endorsement, I've added it to my [long] list of books to buy and read.

Reply

[-]

v-irtual@reddit

This is one of the few books that transfer fairly well to audio - maybe snatch it on audible.

Reply

[-]

BlackFlames01@reddit

Got it, thanks for the advice.

Reply

[-]

caffeine-junkie@reddit

Might revisit that book. Maybe. Tried reading it several (7) years ago, but it was too hard of a read as I was one of several 'Brents" there. It ended up being working then coming home only to read about nearly the exact same thing in my leisure time.

Reply

[-]

heretogetpwned@reddit

I tried explaining that I'm Brent and I shouldn't have 5 managers giving me direction. They told me it's job security. I have 2 interviews this week. :)

Reply

[-]

Sad-Recognition-8257@reddit

If you still have a good relationship with the guy, you better ask him to screen record every critical workflow while you're still talking w him. Of course, he should be compensated for this. Ask him to update the preexisting outdated ones too.

Reply

[-]

xamboozi@reddit

I wouldn't bother trying to do your managers job. Contingency planning is a manager responsibility - sounds like they didn't do their job.

Reply

[-]

shortfinal@reddit

Yep. This is what business insurance is for. Unbelievable how often managers overlook this because they don't value the work their own employees do, just the ability for them to make problems brought to the manager disappear. When we had critical employees leave after weeks, months of squealing under pressure by leadership, I sat my manager down the next day and said "Not stepping in to fill this role. This is your mess to clean up." You know how it's been going around that every accusation is a confession? I'm beginning to wonder if that "remote employees don't do shit" prattling on by leadership was them admitting the same. It was amusing to me how difficult it was to get managers to turn up to critical, recurring meetings involving the team or cross-functional areas.

Reply

[-]

xamboozi@reddit

They offer business insurance for poor planning? That's wild. I always thought if the company was so small they can only afford 1 IT guy, you just hire an MSP instead. You'd get a team of skills at a fraction of 1 IT salary.

Reply

[-]

KeyClacksNSnacks@reddit

His manager shouldnt be a manager anymore unless he tried EXTENSIVELY to get this guy more money to stay and was blocked by higher ups. Way too many managers try to neg their engineers when they talk about or hint about wanting more money and then get Pikachu shocked and still get to keep their job when someone leaves and the entire team suffers due to their loss. SPOF + a lack of serious retention efforts = someone’s head should roll for this.

Reply

[-]

bloodguard@reddit

If he's smart he'd ask for a contract to do all this stuff. Ad-Hoc "helping" a former employer is a legal liability nightmare if you don't.

Reply

[-]

berryer@reddit

I think they meant his two-weeks or whatever heads-up time he gave them, if any

Reply

[-]

Top-Perspective-4069@reddit

He needs a shadow from now until the day he leaves. And you need to figure out what other single points of failure you have and fix those proactively.

Reply

[-]

nitroman89@reddit

Yeah, there was another post like this last week. Someone suggested that they reverse shadow so that they will do his job while he monitors them so that things can get more documentation and find pain points etc etc.

Reply

[-]

223454@reddit

That's how I do things when I leave a job. I spend the first week doing things with someone else shadowing, then the last week I let them "drive" while I watch over their shoulder. By the last few days of someone's employment, they shouldn't be doing ANY hands-on work. Their focus should be 100% on knowledge transfer. A lot of places don't realize that and work that person to the bone until the last minute to get as much value as they can out of them. They don't realize the real value is in the knowledge and experience.

Reply

[-]

Kaphis@reddit

The problem is, organization that have gotten this situation wouldn't have capacity for someone to shadow the leaving person.

Reply

[-]

cszolee79@reddit

hire him back as a contractor for 10\* the money he got as salary

Reply

[-]

charliesk9unit@reddit

Unless the new shop pays him at minimum 50% more, it's never about money for leaving. If you think about it, he'll need to work 50% compared to staying because (1) he already automated the existing system, and (2) he already knows the infra inside out and thus does not take a lot of understanding to get something new done. Bottom line is, it's always because of a toxic environment (to him anyway). People often say you can contact them after they left but in reality, people just don't let the old shop's institutional knowledge lingering in their head. So even if they want to help, their knowledge of it starts to fade on day one he's no longer an employee.

Reply

[-]

isomorphZeta@reddit

> So even if they want to help, their knowledge of it starts to fade on day one he's no longer an employee. What? Is that how it works for most people? It's been nearly 10 months since I left my last position, and I could still get into the environment and fix shit if I needed to. Granted, I worked there for 6+ years, but it's not like that knowledge just evaporates as soon as I take a new job lol

Reply

[-]

j9wxmwsujrmtxk8vcyte@reddit

Then you are wasting energy remembering shit for absolutely no reason.

Reply

[-]

420GB@reddit

Lmao, as of experience and knowledge is ever for nothing.

Reply

[-]

j9wxmwsujrmtxk8vcyte@reddit

Yes, very often it is, especially when the knowledge is of a specific environment instead of general principles.

Reply

[-]

isomorphZeta@reddit

I'm so confused... do you think I'm actively expending energy trying to remember these things? It's not like I'm making a concerted effort to stay brushed up on old environments lmao, I just remember.

Reply

[-]

SecurityHamster@reddit

I always offer to be a resource for quick phone calls for at least a few weeks after.

Reply

[-]

kozak_@reddit

Why? It's a job and they didn't care enough to pay you enough to keep you onboard

Reply

[-]

SecurityHamster@reddit

To not burn bridges. And beside those were all small businesses where it was like family. Been solely in the enterprise the last decade, I wouldn’t make such an offer now.

Reply

[-]

wenestvedt@reddit

Loyalty to the old colleagues, but none for the management.

Reply

[-]

ailyara@reddit

I once left a role for a modest bump in pay simply because there was nothing challenging me at my old role. Didn't want to stagnate. It's not always about the money for everyone. Just my 2p.

Reply

[-]

kozak_@reddit

It's three things : title, responsibilities, and compensation.

Reply

[-]

shadeland@reddit

I once left a role because they didn't fly me enough and I lost my top tier airline status. That's a big deal if you travel a lot.

Reply

[-]

LordLoss01@reddit

Yep. I actually took a 30%+ paycut to change jobs because I realised the job I had was highly specialised and the technology didn't really translate outside that particular organisation. So left the job, took a Service Desk gig for a year before I started moving up.

Reply

[-]

alldefector@reddit

you sure they wouldn't switch jobs for a 40% raise?

Reply

[-]

MrHaxx1@reddit

Absolute bullshit lmao

Reply

[-]

Tireseas@reddit

Fire the idiot management who let one person become such a single point of failure to help offset the cost.

Reply

[-]

Mikeed26@reddit

Ha ha ha this is the kind of thing my company would do 🤣

Reply

[-]

BeenisHat@reddit

ah yes, corporate maff. Gets him off the labor books so they save money!!

Reply

[-]

cszolee79@reddit

Extra end-year bonus and stocks for the mismanagement! :)

Reply

[-]

cpz_77@reddit

lol, we’ve been forced to do this more often than our leadership would probably care to admit 🤣

Reply

[-]

Recent_Perspective53@reddit

So he gave notice or is gone? If he gave notice and hasn't left you, sit down, have him explain and then document. Otherwise start digging.

Reply

[-]

ArcticFlamingoDisco@reddit

Literally block off EVERYTHING from the guy's schedule if he's not gone. Get him to info dump, record it. Do literally nothing except pump him for critical info. 0 Find out his contracting rate. Everyone as a PITA rate, pay it. Expect to replace him with 2-3 people. Get the budget approved for that. It'll take them 6 months to figure out things, and the recorded info dumps help, a lot.

Reply

[-]

RabidTaquito@reddit

Especially record it, but also, be sure to send him drinks or vouchers for drinks or something. Because damn does all that talking for 2 straight weeks suck.

Reply

[-]

CaptSkinny@reddit

Yeah, I was given 8 months notice and offered a generous severance to train my replacements. But I had no say in who was hired to replace me (in a much cheaper job market). When I pointed out they were not up to the task after months of training, I was mostly dismissed as being petty and sowing doubt. When, with 4 weeks left before my departure, it became painfully clear that I was right about their incompetence, my recorded training sessions were the only thing that saved them and enabled the three new qualified replacements to pick up the ball.

Reply

[-]

meatymimic@reddit

This right here. Every system you document now is going to be several sleepless nights and stressful calls saved later. This type of situation is how you get "so and so coded that and we dont know how it works so no one touch it" systems that we like to make jokes about. Every last hour of this guy's 2 weeks needs to be spent with someone who is writing down how shit works.

Reply

[-]

notHooptieJ@reddit

and the first step upon his replacements arrival needs to be replacing and or documenting. usually its toothpaste you cant put back in the tube. Its the reason "the new boss" always gets a bad rap for ripping out systems on day 1. If you cannot support the system you CAN NOT RUN IT.

Reply

[-]

meatymimic@reddit

You shouldn't run it. But yet - folks do.

Reply

[-]

notHooptieJ@reddit

the folks still there or leading via attrition will. but you bring in an outsider, and that WILL be #1 on their list. (its not only good form to kill the bad thing, but they get a big resume buff walking in 'converted *archiac system to new whizzbang*' on their first big project, and can/will use it to springboard right on out of there.)

Reply

[-]

fencepost_ajm@reddit

Recording, not just writing down. Draw pictures on whiteboards while talking things through. Do the writing as well, but record.

Reply

[-]

scriptmonkey420@reddit

Were I work that is 90% of the legacy business applications. They are in maintenance mode and are only kept limping along. We ask when they will be modernized and they just laugh. It's frustrating when the same teams will complain of performance issues and blame SSO....

Reply

[-]

Suspicious-Line-5126@reddit

Bruhhh, no need to literally get so excited. It is not your startup

Reply

[-]

ArcticFlamingoDisco@reddit

I get hired because of my rep. My rep is solid because I do a professional job as long as I'm collecting the paycheck.

Reply

[-]

PinotGroucho@reddit

This right here. Violently purge the delusion from your system that you're going to find "the one guy" that will carry his workload with the same competence level at the same price point. You struck gold without realising it and the vein has run dry. Treat him like a guru whose guidance cannot be missed and his ego and decreased operational responsibilities might keep him around on a retainer or however you need to keep him. Hire at least two or preferably three specialised engineers to quickly get up to speed under his guidance on their specific fieland maybe you keep ahead of this thing and possibly take the opportunity to adapt and prosper. Fail to do this and in 12 months time all the automation, not being maintained with the same vision that saw it realised in the first place, has now turned into deferred maintenance. Give it another 12 months and it is now technical debt that will cost an arm and a leg to update/replace. The lackadaisical approach that seems to permiate your question to this looming catastrophe is not very encouraging to the future survival of your startup.

Reply

[-]

cgimusic@reddit

> Expect to replace him with 2-3 people. 100% this. The company is only in this situation because they were too cheap to pay more than one person to do a role that is apparently critical to the functioning of the company.

Reply

[-]

kozak_@reddit

I can't imagine if they gave him a role title change and paid him 15% more than his new position that he wouldn't stay?

Reply

[-]

fencepost_ajm@reddit

**Do this before he forgets** Even if you're able to call him back at a consulting rate to answer some questions he's not going to be spending his time immersed in systems for your company. The longer he's been away the more it's going to be "well, I think it worked like xyz but I don't really remember." Asking questions like "ok, process 123 just stopped and we're not sure why. Let's brainstorm OUY LOUD and on a whiteboard the troubleshooting steps, process flow and what we should look for at each stage."

Reply

[-]

FatuousNymph@reddit

6 feels optimistic

Reply

[-]

OstrobogulousIntent@reddit

When I gave 2 week notice on a job where I had been keeping a LOT of plates spinning in the air at once.. I warned my boss - "look, I've thoroughly documented things but you might want to block off time with [list of people here] and let me brain dump on them so they can ask questions" Boss didn't get around to it till my second to last day despite my repeated insistence they'd want more time. TL;DR: I did my best to give them as much brain dump and help as I could in my last 2 weeks but the head of IT squandered almost all of it. I was happy to be out of that place cuz the head of IT was a total dundermuffin.

Reply

[-]

Blackpaw8825@reddit

Or, you do like my company did and fire the guy then ask around about what he was doing. I hate it here

Reply

[-]

I_COULD_say@reddit

I was the guy that gave notice and, instead of spending that time as an info dump / hand off, they accepted my resignation right then and there. Idiots.

Reply

[-]

shitlord_god@reddit

I love it when this happens, grow your legend.

Reply

[-]

I_COULD_say@reddit

The best part was being able to watch their productivity drop in real time. They have to submit reports on what they've done / accomplished every quarter. One big metric is how many tickets are closed. Those reports are public. The quarter after I left, their count wasn't even half what it was while I was there. Oh and they had to add a role to cover my workload. They could've just given me the raised I asked for but they did not. Again: Idiots.

Reply

[-]

shitlord_god@reddit

I'm always shocked by how much that forced error happens and cuts at the efficiency of capitalism due to that unwillingness to compensate someone appropriately, but the continued use of it as a practice.

Reply

[-]

I_COULD_say@reddit

Yep. I asked for a fraction of what the new role cost them lol

Reply

[-]

Blackpaw8825@reddit

I left my old boss with a 37 page document to jump start successor with indexed and flagged guides on all the home-grown tools I had made to make the job easier. It was all notation and screen shots, you needed to just click the things I showed you that way my old boss could divide up the things I did day to day between whoever needed to do them. 2 years later my employer merged with them, and the person who took over after my old boss left that team was sent my way for help since I'm "good at that kind of thing". She had bungled EVERYTHING, all the outputs were unusable, they'd lost business over it, and casually dropped 'my predecessor left this, but it's huge so I never read it, maybe it'll mean something to you' I was PISSED. That was 3 years ago and I still never "figured it out" so she's got herself and 6 people doing it all the hard way 40-50 hours each, instead of using the tools I left her which a single person could do every Monday, with maybe a bit of time Tuesday if I slaked off.

Reply

[-]

I_COULD_say@reddit

I couldn't get my team behind the simplest of things. "Stop logging into a DC to unlock user accounts. Install RSAT on your machine and do that, instead." That was a problem. PowerShell? NO WAY JOSE YOU CANT TRUST THAT Anyway, I left that place in better shape than I found it, so that's a win for me i guess.

Reply

[-]

Shazam1269@reddit

And, we don't need to replace him/her. Soak up those extra duties you ungrateful maggot!

Reply

[-]

shitlord_god@reddit

"Just use AI to streamline your workflows"

Reply

[-]

JPowTheDayTrader@reddit

This triggered me

Reply

[-]

shitlord_god@reddit

The truth of our institutions Comrade.

Reply

[-]

JuanPabloElSegundo@reddit

Is it also your fault when you can't support his work too?

Reply

[-]

TU4AR@reddit

Ayyyyyyy We are just letting you go now, thanks for the two weeks notice. Don't really need anything from you.

Reply

[-]

Capt_Blahvious@reddit

I was this critical-infrastructure-guy at a company and they paid me my fuck-you-hourly-rate for 5 weeks to get my info dump of how things work.

Reply

[-]

ledow@reddit

Sounds like you should have been on top of documentation the whole time. And if you weren't... maybe today is the day to start getting on top of it, while the guy is still around. And then maintain it for everything else, including the stuff that that guy has nothing to do with. Documentation - the one thing that EVERYONE wants to be in-place, complete and accurate and the one thing that NOBODY EVER BOTHERS TO MAKE SURE is in-place, complete and accurate.

Reply

[-]

motherfuckinwoofie@reddit

I realized it's easier for me to sneak off and play on my phone than it is to sit at my desk and create/update documentation. It's like if I'm off in the plant, then I'm busy, but if I'm at my desk, I'm just playing. So that's why I take time every day to sneak off and job hunt.

Reply

[-]

Tetha@reddit

I've also been pushing: Always be ready to refine documentation. Our documentation is not the Holy Bible, it's a beat-to-shit notebook with coffee, oil and other stains. If something helped you, put it in there. Like sometimes, when I note down what I do it's like "Step four: reconfigure && failover postgres according to procedure". It's enough for me and another person on the team. In some of our runbooks, my one-line bullet point is now a 7 element checklist and 12 more steps in the runbook. That's great, because it's very complete at that point. But if it is not enough for you, please expand this as necessary. If you are not sure about your expansions on the documentation, setup a call with me. Discussing mostly-correct, but subtly wrong understandings of systems and procedures is often a very valuable thing to do. But this results in very effective documentation. By now people either resolve even huge and complex outages and then discuss weak points in the documentation, or I get called to confirm that we're indeed very fucked at this point. It's good to know that it'll be kept running even without me.

Reply

[-]

LordLoss01@reddit

I'm bit weird with documentation. Either I'll have none at all or it will be so detailed that a ten year old could follow it. Admittedly, that's lead to a lot of my colleagues not reading it because of how long it is. Still comprehensible as I treat the reader like an idiot; the document is literally foolproof.

Reply

[-]

ledow@reddit

Yep, the purpose of documentation is not to train any idiot to do your job, it's just not possible. It's to provide that one vital line about how (and often why) something is setup in a particular way.

Reply

[-]

iliark@reddit

The selfish thing to do is to not document as an incentive to not get fired. Only works at small organizations though where one person can make up a significant percent of the responsibilities.

Reply

[-]

RikiWardOG@reddit

this issue isn't selfishness many times, it's time. You aren't given the time to document things. Besides, documentation only gets people so far. People fail at the most basic critical thinking when reading through documentation if you can even get them to look at it in the first place. Not to mention, things change. A lot of times documentation is already outdated by the time you finished writing it.

Reply

[-]

Centimane@reddit

Also the smaller the team the less impact documentation has. Sounds like the person in question was working independently. They wrote some documentation, but it sounds like nobody needed the documentation, so it fell by the wayside. If they're the only one working on something, the documentation is only for them or for when they leave. It's also more practical to answer questions directly if there's a small enough audience. The nice thing about documentation is it scales well. Need 100 people to know an answer? They can all read the doc. Only 1 person needs to know the answer? Just telling them the answer is faster - though not better.

Reply

[-]

FarToe1@reddit

Exactly. When you're siloed like this, you can write all the documentation in the world, but unless your team knows where it is and what it actually means, it's useless. And once you realise that, it can be properly hard to stay motivated and write it at all. Plus - the overview plan is just, if not more, useful, and that often gets forgotten by everyone who assumes someone else is in the loop. Good management is the key, and I think knowledge and change management is genuinely one of the hardest things to keep on top of over time.

Reply

[-]

dathar@reddit

You are usually in a time crunch but at least write down notes and general design docs/ideas for yourself. Works as a refresher to yourself when you're a couple years down the road and something dies and be a lifeline for anyone that might have to poke under the hood. Something like # Service copy groups and memberships from IDP to _____ * Service runs on VM _____ * Runs hourly * Uses API key from IDP. Current name of token is group\_copier\_service. Put API key at ____ in the credentials section if regenerated or revoked. Link at <a href> * Script lives on fileshare/git. <a href> Hopefully enough people can at least use it as a starting spot or be a refresher when Security goes willy-nilly and starts killing stuff.

Reply

[-]

mschuster91@reddit

Often enough you don't have the time to properly document stuff simply because of a lack of staff.

Reply

[-]

iliark@reddit

True enough. I document because I'll forget what I did if I've gone to sleep, eaten a meal, or someone has talked to me since the last time I've thought about it.

Reply

[-]

motorik@reddit

Selfish? After 20 years of being considered a consumable like post-it notes or copier paper, I operate from a standpoint of total self-interest.

Reply

[-]

iliark@reddit

Being selfish isn't inherently wrong, it's just self-preservation. Gotta do what you gotta do.

Reply

[-]

MyStackRunnethOver@reddit

One person can make up a significant percent of the responsibilities of a team, department, or org even at FAANG companies

Reply

[-]

NoPossibility4178@reddit

I mean when I leave it's not my problem :D sucks for the coworkers but if the company wants me to have 0 time for documentation, I'll spend 0 time documenting.

Reply

[-]

thunderbird32@reddit

Documentation is, unfortunately, the first thing to go by the wayside when overworked admins are pushed to take on more and more work. One wonders if this may be why the guy is leaving?

Reply

[-]

ledow@reddit

I agree absolutely, don't get me wrong. But that's even more reason that SOMEONE ELSE should have been making sure his documentation was being written, checked and updated, and him being given time to do so. The guy who's working miracles isn't the only one who should be ensuring the documentation is up to date, or else stuff like this happens. It's like having backups and then the "guy who does the backups" leaves and the company doesn't know where / how to access their backups - or if any even exist or were ever checked at all. You need OTHER PEOPLE to be involved to make sure that works out. Usually the people to whom you would expect it to fall if said guy leaves.

Reply

[-]

Fitz_2112b@reddit

The last days of this guy's employment need to be writing documentation for everything that he's done. Also, as long as he's leaving on good terms, consider trying to convince your management to get a consulting arrangement set up with him so that you could call him in if needed for help

Reply

[-]

EndPsychological2541@reddit

He gave his notice a few days ago.. You're talking like he's been gone months.. Is reddit just a sea of fake stories now?

Reply

[-]

AwalkertheITguy@reddit

Yes, sir. It is actually.

Reply

[-]

dabbydaberson@reddit

Welcome to the find out section of tech debt! Brought to you by zero documentation and over worked engineers.

Reply

[-]

the_one_jt@reddit

So there is an opening?

Reply

[-]

Professional_Mix2418@reddit

Go to the repository and use clause code and do a /init and after that ask for the questions. I mean you did ask him to check in the code didn’t you?

Reply

[-]

420GB@reddit

Just dig through it and understand it? Learn, document if you have to. Sounds like a great opportunity to get smarter.

Reply

[-]

ArchivisX@reddit

Claude Opus 4.6 would have a field day with this.

Reply

[-]

HODL_Bandit@reddit

If you treat your workers nice and friendly they would never put you in the shitter. Your fault.

Reply

[-]

1z1z2x2x3c3c4v4v@reddit

> how do you recover when everythging is basically locked inside the automation stack of an employee who just left? 1. You never let it get this far, and require documentation for all the parts. Good documentation so it can be rebuilt by someone else (with the required skills). 2. Find a good consulting company to assess and review everything as it was designed and built.

Reply

[-]

Oflameo@reddit

Either learn the system or offer it to someone who willing to learn it.

Reply

[-]

budtske@reddit

Step 1. Aquire time machine and give him a raise and better work life balance. I've seen it before, I've actually lived it. It involves a whole lot of pain and commitment from employees and if the situation is bad enough like mine was lots of people will drop off because of it. Fast tracked promotion opportunity, sure. Not worth it in retrospect.

Reply

[-]

t_whales@reddit

Sounds like he set himself up as a consultant quite well. He’ll be making double per conversation

Reply

[-]

asdlkf@reddit

double? try 10x.

Reply

[-]

binaryhextechdude@reddit

If he's charging double he's doing himself a disservice.

Reply

[-]

excitedsolutions@reddit

Get him to sign a $150/hr engagement with the existing org. Then, utilize reaching out to him when you are lost when trying to replace the existing functionality.

Reply

[-]

asdlkf@reddit

If he/you are on good terms, offer him an ongoing time-and-materials contract at 5x his previous rate on 1-hour billing increments. Just because he is "leaving" doesn't mean he died or something.

Reply

[-]

gramathy@reddit

This is the cost of automation. You can run more lean but you need someone who understands the automation system to keep it working, which means you still need "bus redundancy" for every system or your organization is at risk.

Reply

[-]

WindyNightmare@reddit

This is called opportunity. Time for you to shine.

Reply

[-]

clbw@reddit

Where I work cross training is the rule I’ve been with this place for 10 years. The first two years. All I did was work with other engineers learning how to do their job back-and-forth up and down now I’ve done the same for people that have started in the last eight years. It’s the only place that’s ever done this that I’ve worked up.

Reply

[-]

yycTechGuy@reddit

A good AI agent will give you great insight into how everything works in less than a day.

Reply

[-]

jpenriq1@reddit

These situations are beyond laughable. It was always just a matter of time and still, no one did anything about anything. Burn baby burn.

Reply

[-]

phileat@reddit

“Especially when I have questions” - it’s quite unfair not to continue compensating the guy if he will help with your infra.

Reply

[-]

LordLoss01@reddit

Meh, I do that as well for places I've worked at. I don't do it because I like the business. I do it because I like my former colleagues and some of them are actual friends who I don't want to see struggle.

Reply

[-]

iama_bad_person@reddit

We had this happen to us once, years ago. Screwed us big time, took me around 6 months of digging, diving and poking around our Infra to finally understand how it all worked. Now we have "Read-only Fridays" where we document everything we didn't document during the week. Go back over code with another SysAdmin and add small comments to the code where needed. Add an entry to our documentation system about what the code is and why it is needed. Edit our "master" Visio file and visually pop in where the code lives in the grand scheme of things, what it connects to, what it writes.

Reply

[-]

Tricky-Service-8507@reddit

Sounds like a pitiful excuse for management

Reply

[-]

drifterlady@reddit

Ask if there are any other jobs at his new place quick. Or, become the system expert with your new found knowledge and then retire to be a contractor.

Reply

[-]

BadSausageFactory@reddit

we had one of those, three years later and we're still digging ourselves out of the silo. it's a hole, really. they call it a silo because that sounds like you accomplished something but no, you really just took all your knowledge and buried it in a hole.

Reply

[-]

InformationRound3874@reddit

I have heard so many people talk about and dream about being the guy leaving in this specific situation. “If I left….” I used to think that way. Now I’m old and realize that everyone is replaceable in the end. The greatest employees are the ones who document so much that the company doesn’t need them.

Reply

[-]

neale1993@reddit

Does he have a notice period, or has he just *gone*? Not an ideal place to be in. Everything needs to be documented, from the ground up and should have been from the start. How it works, what it does, everything. If there is someone else in the same team as him, there needs to be some time to handover.

Reply

[-]

cryptme@reddit

Problem is, this type of employee is put to work to solve things. Management is happy as he is drowning in things to solve. The more he solves, the more is thrown at him. Expect him to fight management is nonsense. This is clearly a management problem.

Reply

[-]

BrainWaveCC@reddit

>Ops engineer who built half our automation just gave notice. Nobody understands the system Ooops... [](https://www.reddit.com/r/sysadmin/?f=flair_name%3A%22Question%22)

Reply

[-]

flummox1234@reddit

You should read the Phoenix project https://itrevolution.com/product/the-phoenix-project/ Also just start reverse engineering all his automation into documentation. I hate to be the one to say it but AI could probably be a godsend here if it's just simple scripts as it tends to do pretty well with those IME and you can have it write some documentation. No matter what you do step one is make sure the scripts are in version control.

Reply

[-]

Id_Rather_Not_Tell@reddit

Documentation is the only real answer. You might want to leave the door open to bring him back on a contractual basis to assist with that.

Reply

[-]

ride_whenever@reddit

I’m going through this at the moment, with a three page word doc for handover. I’ve been pulling metadata wherever possible and using cusor/claude to decipher it, which is fine, but definitely needs a load of assistance and lacks business context. Even with being hooked into all our onboarding documentation. If you’ve got time, beyond the docs, you want to know what he’s still doing manually/fixing on fallover, as well as what he was planning on doing next, and what bits he wanted to redo/replace

Reply

[-]

TireFryer426@reddit

Try to get copies of all the scripts in one place and then ask claude code to review all of it and create operations guides and a high level overview of what everything is. Should be able to at least get your legs under you at that point.

Reply

[-]

sriracharade@reddit

Backups of hardware-- Good Backups of people-- Bad Never fails to amaze me. Not you, OP. I'm sure you had nothing to do with this, but someone should have planned for the possibility of him not being there for whatever reason.

Reply

[-]

jasondbk@reddit

If the person documents their work and does a good job of it that should be part of the change management process. I had a boss who didn’t know anything about programming. My first project I documented so well they could understand what and how the program worked. It was then reviewed by a senior programmer. As time went on the programmer reviews went quicker but my immediate supervisor still read the documentation as if her job depended on it.

Reply

[-]

awesomenessjared@reddit

A beautifully-crafted, karma-farming spam post. It appeals to everyone here because it's a dream scenario for the guy leaving, and it's fun to make fun of "you" for being an idiot here. I cannot believe this post is sitting at 95% upvotes right now; /r/sysadmin is rapidly falling apart as people eat this slop up...

Reply

[-]

grahag@reddit

How much would you lose if that system were to go down and not be restored? Double that and offer it to the engineer.

Reply

[-]

thenewtnik@reddit

Sorry but it's too late for that type of reasoning. At best, you'll just be kicking the can down the road.

Reply

[-]

Generic_Specialist73@reddit

It sounds like your company made a big mistake by letting someone so valuable leave. You should have had golden handcuffs on him.

Reply

[-]

DilithiumCrystals@reddit

I still haven't seen a comment where some recommends getting an enterprise grade Workload Orchestration tool. I work for a company that makes one and I am on the migration team that converts from other systems to ours. If you are coming from a collection of bespoke tools, some serious forensics will be required, but it won't be impossible.

Reply

[-]

sammavet@reddit

No documentation? See what dude remembers about what scripts/automation and document it all!!!!

Reply

[-]

Scubber@reddit

Here is the enterprise managers playbook: Hire 20 of the cheapest offshore workers you can find and see if one of them understands any of it.

Reply

[-]

Odd-Assumption-9521@reddit

QOL huh? Next time, destroy the employees QUALITY OF LIFE covertly and prevent them from leaving, trust me!!!!

Reply

[-]

Odd-Assumption-9521@reddit

The best way to keep ur talent is to accuse and smear campaign them so they stay urs, humble them, mess with their head, lungs etc

Reply

[-]

talexbatreddit@reddit

I would hire someone smart on contract immediately, and have them shadow this person, documenting as they go. Unless you have someone permanent who is capable of doing the job. And I'm guessing that project (Understand Dave's Work) is the highest priority project right now.

Reply

[-]

deacon4@reddit

I cannot tell you the number of times I've been contracted to come in after someone who automated everything left, to figure out what they did, how they did it, document it, and update it. I actually love doing it! I learn so freakin' much on how different people did different things and why. If you can, try it! Learn what he did. Reverse engineer it. Do you know what language(s) he wrote most of his automations in?

Reply

[-]

oni06@reddit

I hope all the automation is in a git repo somewhere. If so what I would do is have Claude scan the repos and give a summary of what everything does (ie it will document the automations.) I think it’s a good place to start at the very least.

Reply

[-]

hijinks@reddit

so i know a lot of people hate AI/LLM.. get a claude code account for $100 a month and have it analyze everything that does it and have it write docs about how it works I'd had it break down home grown code that was internally developed and people left and i was able to pick it up in 30min to start to commit where before it might take be 5-10 days to grok it all

Reply

[-]

Delicious-Squash6327@reddit

This is a good option. We have used it to document and reverse engineer poorly documented code.

Reply

[-]

TechGuyworking@reddit

How do you know it's accurate? My recent experience with AI chatbots is that they are confidently wrong at least 20% of the time.

Reply

[-]

alluran@reddit

20% wrong, or 100% missing.... I'll take 20% wrong. Can't be worse than most documentation which is 50% out of date :P

Reply

[-]

hijinks@reddit

chatbots are vastly different from how something like opus 4.6 can analyze code.. I've had it have R/O access to DBs and figure out what queries were causing high I/O times. I've had it figure out and fix issues with very complex ci/cd. I'd say people screw up far more then opus 4.6 does.

Reply

[-]

ITaggie@reddit

> wrong at least 20% of the time. Honestly sounds about right for hand-made documentation too lol

Reply

[-]

AloofGamer@reddit

You can pretty easily run tests against its understanding to validate results

Reply

[-]

What-a-Crock@reddit

Or hire an acutal dev. So Claude doesn’t [ruin everything](https://www.tomshardware.com/tech-industry/artificial-intelligence/claude-code-deletes-developers-production-setup-including-its-database-and-snapshots-2-5-years-of-records-were-nuked-in-an-instant)

Reply

[-]

alluran@reddit

Claude didn't ruin everything. An incompetent employee did. Holy shit, so many misses on that one.

Reply

[-]

FauxReal@reddit

Yeah it's being targeted. [https://www.techradar.com/pro/security/claude-desktop-extension-can-be-hijacked-to-send-out-malware-by-a-simple-google-calendar-event](https://www.techradar.com/pro/security/claude-desktop-extension-can-be-hijacked-to-send-out-malware-by-a-simple-google-calendar-event)

Reply

[-]

billyalt@reddit

Christ

Reply

[-]

Arudinne@reddit

Claude actually advised against doing exactly what the devloper was doing, so there's that.

Reply

[-]

Abysuus@reddit

Giving it read vs write/modify are vastly different use cases.

Reply

[-]

Palaver999@reddit

Upvote for grok.

Reply

[-]

thunderbird32@reddit

Yes, because letting the CSAM generator loose on your data is a good thing... /s

Reply

[-]

octave1@reddit

Can it deal with a dumpster fire of spaghetti code ?

Reply

[-]

climb-it-ographer@reddit

Yes. Especially if you start writing tests as a part of the process.

Reply

[-]

FragileEagle@reddit

\+1 on this. people talk a lot of crap about claude but this seriously saved me in a similar situation

Reply

[-]

cpz_77@reddit

I’m wondering if is a situation where AI may be able to help significantly - start recording some calls where he presents and goes into detail about how this stuff works. With the transcription features, and possibly AI help, turn that into documentation. When possible, have AI crawl structures and document them itself - like databases if he maintained those, folders full of scripts he wrote, etc. It’s ironic I’m recommending this since I just posted another post about a bunch of concerns I have with AI lol, but for this type of scenario I think it’s a use case where it could actually help. If I’m off base and there’s some reason this obviously won’t work, then disregard the AI part. In any case though, I would wipe everything else off the guy’s calendar and schedule knowledge transfer meetings for as many critical topics as possible and be sure to record them all (and set the recording not to expire if you’re on Teams, otherwise by default it may disappear in 60 days), and ask him to start writing up notes on anything and everything he can’t, starting with the most important stuff first. Hopefully he may have a lot of such notes already and may just be able to share them as-is or with minor adjustments. If not, and if AI can’t help, then that could be a ton of work, but it’s critical to get as much as you can while you still can. Also maybe arrange some agreement so you could bring him on to consult for some hours if needs here or there, if he’s willing and both sides can come to an agreement.

Reply

[-]

isomorphZeta@reddit

Your manager needs to do the following: 1. Find out his contracting rate. 1. Pay him to step y'all through how everything works. 1. Document everything. If he won't do work for y'all on an hourly basis, or the company can't/won't pay him, then your life is about to get a lot more difficult lol If people aren't required to document processes and procedures, they usually won't - either to save time, ensure job security, or both. That's on your manager for not enforcing better documentation policies, and now the company is going to pay for it with time and money. Good for your Ops Engineer, though. If the company let him get away with minimal to no documentation, he just got a nice raise ***and*** may get some fat checks from the company to put a nice bow on everything on his way out.

Reply

[-]

ComfortableAd8326@reddit

Claude-code can make incredibly light work of documenting everything (using a self-hosted model for obvious reasons). Ideally your engineer would do this themselves before they go so they can sanity check everything

Reply

[-]

Distinct_Goose_3561@reddit

If possible, work with him to document everything. If he’s gone- this is where LLMs are actually pretty great. As long as you can feed it all of the various scripts and provide basic guidelines you’ll get pretty far. It won’t be perfect of course, but it’s far better than trying to recreate this by hand.

Reply

[-]

Individual_Hair1401@reddit

In the current 2026 infra landscape, "Tribal Knowledge" is the single biggest point of failure. If your automation is a "black box" that only one person understood, you aren't actually automated; you're just brittle. Real talk, the first thing you need to do is a **Permissions Audit**. Ngl, the number of times an ex-employee's personal Github token or a hardcoded SSH key is still floating around in a production script is terrifying lol.

Reply

[-]

kerosene31@reddit

Other than the obvious "don't let things get here". Nothing should be considered "done" until the documentation is done. Another obvious answer - hire someone else. Start with what is running and when. What happens when something doesn't run? Your biggest fear immediately shouldn't be something big breaking, but something simply stops running and nobody notices right away, but it has big impact down the line. System A and B now don't have the same data and the quarterly reports are all screwed kind of thing. What is/was monitoring all this? The details of what's happening are for later. The first question you need to answer is "X didn't run last night" - what does that mean to everything else? The first thing I would want is a one or two page document of what runs and when. (and the "what" can be vague until you have more time).

Reply

[-]

Arts_Prodigy@reddit

I’m not blaming you although I think at some point in your career everyone needs to “manage up” a little bit. But frankly it’s a shame that this is/could still even happen. Unless the engineer is secretly writing everything in something like whitespace. There’s no real reason this should happen. The same process that stop juniors from bricking production when making a small change should be applied to the senior level engineers to ensure shit like this doesn’t happen. This is largely a process issue though there’s only so much a curious mid level can understand tbh. At no point should there be so few seniors and so little documentation that issues can’t be easily resolved or architecture easily understood.

Reply

[-]

hajimenogio92@reddit

This was me at my last job. I built the majority of their infrastructure in AWS, wrote most of the IaC/pipelines/automation, and was a one man team for a smaller start up. I spent my last week just documenting and working with the leads for outstanding items and their progress. Set up time with him and document/record as much as possible

Reply

[-]

Natirs@reddit

>how do you recover when everythging is basically locked inside the automation stack of an employee who just left How much money are they willing to pay you? Up front and in writing. And you don't do a dime of any of it until it hits your paycheck.

Reply

[-]

ProfessionalEven296@reddit

What’s your normal job? Because to handle this (without a consultant) will take a lot of time. You can’t just do a bit here and there; you need to be assigned solely to this task for a while.

Reply

[-]

182RG@reddit

…”and I just realized”…

Reply

[-]

clef75@reddit

AI has changed the game on this. Claude code can figure it out and write up docs.

Reply

[-]

Major-Astronomer7529@reddit

Documentation you'll likely need: 1. Flow charts, with labels, for each automation 2. Service accounts used for the automations, if they're attached to the Ops person they stop working when account is disabled 3. Passwords for service accounts 4. Contacts for each related application 5. Documentation related to what/why something was implemented 6. Do your knowledge transfer sessions in Teams/Zoom and transcribe them 7. Ask for any "gotcha" thinks the Ops person can think of; first list them out, then circle back and discuss how to deal with then 8. All vendor contact and contract information 9. List of all technologies used in the automations, should be part of the flow charts... 10. Where API keys are stored and who has access I'm probably missing some stuff but this is what you, or the next person, will need to maintain this environment.

Reply

[-]

donjulioanejo@reddit

Honestly? You don't want to own this shit. Tell management to hire a consultant to figure it out. Or to retain the guy as a part-time advisor with a fixed retainer.

Reply

[-]

Loud_Posseidon@reddit

At this point in time I would just dump running processes to Claude as a starting point and guide Claude to explain everything. Where does the config come from, what happens if this process dies, how to get network connections, what do they point to, what’s running over there, really the basic stuff. Bet you it’ll understand and explain better then the guy ever could. At the fraction of cost. Source: just had similar situation

Reply

[-]

frankwiles@reddit

Best advice I can give is hire an expert or dev agency to come in and dig through everything. Documenting and cleaning up as they go while you’re hire the replacement. They can the train and help the replacement during a transition. And then learn from this so it doesn’t happen with other key staff.

Reply

[-]

glotzerhotze@reddit

I just spent a year cleaning up such a situation. It‘s always fun to walk into a project and reverse-engineer someone‘s code, but it takes time to do so. If you still got access to the co-worker, get as much information as you can. Or hire a capable consultant to help you figure out $stuff.

Reply

[-]

mrbiggbrain@reddit

I would lock out their access immediately. They are no longer allowed to touch a keyboard or do any work. Everything they need done someone else does and records the screen while they do it. Every minute of their day they are on a meeting with someone doing handoffs. Do not let them get pulled into "This has to get deployed before Steve leaves" you don't have Steve you have Steve's ghost, boo, scarry! Someone else does everything, everything, I mean it you don't kick the can and say "We can figure it out later, have Steve do it now", every last thing is done by someone else. If you need to tie his hands behind his back to stop him doing work do it.

Reply

[-]

wwb_99@reddit

Sounds like a great way to make the guy say "you know I have a new job and better things to do, I'm not sitting through this bullshit."

Reply

[-]

Dry_Inspection_4583@reddit

First off you need management to recognize that they directly caused this by lack of evaluation and adequate compensation. For your side, yah, start from the bottom and document everything. A wiki if you want, or a notebook, but write everything down and work from one system upward.

Reply

[-]

progenyofeniac@reddit

Hire someone who’ll cost you far more than the person who left. Someone who can not only create new automations, but who can understand and reverse engineer the current ones. As someone who’s been the one leaving in the past, reevaluate your pay and overall compensation for the role. Consider whether it might be worth hiring two people in this role to increase your “bus factor” and share knowledge. Possibly conduct a frank exit interview to get this person’s input on what would have needed to be different to retain them. It frustrates me to no end to see a business that seems to have never considered what it would do if one person left. I worry about the overall business plan and almost feel forced to assume a bit of tone-deafness to employees’ concerns. I hope this can be a learning experience rather than simply the beginning of a trend.

Reply

[-]

vendeep@reddit

I will get an ear full for this, but documentation isomething I can trust these AI agents to do well. Dump his code base into there and have them document it. Assuming he hasn’t left yet, you could start the knowledge transfer.

Reply

[-]

eufemiapiccio77@reddit

Good on Dave. There’s that post. If Dave leaves you might as well shutdown the company

Reply

[-]

The_Wkwied@reddit

Bus Factor. If somebody getting hit by the bus would cripple your org and wipe out knowledge, you MUST get to work on documentation. If the guy is still there, bring up the case with your/his boss that this fella spends his two weeks making KBs that are el5 level easy to understand. If the guy isn't there, bring the case up with your/his boss that you DO NOT KNOW what to do **WHEN** (not if, when, if he has been maintaining these for years, it is WHEN) things break.

Reply

[-]

llDemonll@reddit

Hire a new engineer to take it over and modernize / centralize the processes. Don’t take it over and don’t have your team take it over.

Reply

[-]

toebob@reddit

First, spend some time recognizing the mistakes that led to this point. Nobody should be irreplacable and you should have had plenty of documentation and cross-training on all aspects of your production systems. This isn’t an “I told you so” or blame exercise. It is a lesson learned and action item to correct moving forward. For your current systems, I’d recommend dedicating resources to code review and documentation. Those resources could be time from existing employees, consulting hours, or a new hire to take the place of the person leaving. In business terms, your DR risk just went up. The likelihood of a failure has not gone up but the impact has. You no longer have the expertise to fix your own system if something fails. That translates to dollars, which the leadership should understand.

Reply

[-]

mortsdeer@reddit

Just wanted to amplify the new hire idea. Updating tests and docs can actually be an excellent way to learn a new (legacy) system. It actually works best with mid to senior hires, since they're less likely to take the existing code or docs as gospel.

Reply

[-]

fresh-dork@reddit

spend the next week and change getting him to only draw diagrams of how everything fits together and walk you through common/critical tasks - you type, he speaks. get his top 10 list of important shit that requires care or can break badly, go over metrics and make sure you can log into every damn thing he's got set up

Reply

[-]

redunculuspanda@reddit

Probably time to start thinking about an integration strategy. Someone will have a fun project reworking all this stuff and migrating it to an orchestration platform.

Reply

[-]

Luckygecko1@reddit

Being proactive is how to deal with this. I know that's too late for you now. Assuming you can deal with the security risks, AI has become really good at 'tell me like I'm five' how this code works. You need to do an audit to make sure you have full control of all these automations. It my prior location, every systems engineer was paired with a partner and forced to share the same office. We constantly cross trained on our work. If one of us was out of the office the other was expected to be able to give a brief on the other's work. In addition, we had a 'they were hit by a bus' folder in the IT firesafe. It was in a tamper seal, updated with current critical passwords and other authentication I would use.

Reply

[-]

Squeezer999@reddit

This isn't your problem. Let management figure it out and come up with a solution

Reply

[-]

everflowed@reddit

Companies must sometime understand that they should not have people that are SPOF.

Reply

[-]

Code-Useful@reddit

If no one can handle it it sounds like you're hiring someone who can

Reply

[-]

tuvar_hiede@reddit

Sounds like a management issue. I wont let anyone on my team implement anything that can't be replicated by thier peers. Its just begging to gave this happen.

Reply

[-]

InfraScaler@reddit

Time to splurge in a few Claude/Codex/Copilot subscriptions

Reply

[-]

ClayDenton@reddit

Is he still working? It might only take a couple of days to get this informally documented. A good way to do this is to set up a series of teams meetings with folks around the team and in those have him present the workflow of specific automations along with his development flow... And record the screen share / meeting. And have them ask questions. This will make it so much easier

Reply

[-]

wrt-wtf-@reddit

One day at a time. People really need to think a lot more about this. As the market refuses to increase wages people with very good experience become easier to pull away from somewhere that the employer treats them as rusted on. Even in a time of AI those people are more valuable because the experience they bring is a force multiplier and a surer bet than just AI alone.

Reply

[-]

tarvijron@reddit

![gif](giphy|cnbcRzMm86PLzEAQko) Ask your leadership how hiring is going and then kick fucking back.

Reply

[-]

jnsh7@reddit

I might be missing something, but if it were me I’d probably start by figuring out what access exists and making sure the company actually controls it (and knows how to access). Repos, schedulers, servers, service accounts, API keys, dashboards, etc. Basically check there’s nothing important tied to his personal accounts. From there I’d build a quick inventory of the “tons of stuff” you mentioned and group it by criticality/impact so you know what actually matters if it breaks, and where to prioritise discovery. Then I’d focus the knowledge transfer around those. Have him walk through where the code lives, where logs are, what usually breaks, and how to run things manually if needed. I’d record or AI-transcribe those sessions if possible. Realistically you’re probably going to lose some knowledge and won’t understand everything straight away, but if you know what exists, have access to it, and know what’s mission critical, that’s already a solid starting point. Good luck

Reply

[-]

Master-IT-All@reddit

Create documentation. This is pretty easy now, just take the scripts/automation and drop them into a licensed AI like Copilot to analyze and document.

Reply

[-]

stacksmasher@reddit

That's a leadership problem.

Reply

[-]

ColoradoStudent@reddit

Better fire up Antigravity and start reviewing the code while he's still around to ask questions of.

Reply

[-]

Proper-Cause-4153@reddit

It's frustrating when engineers don't acknowledge that documentation is part of the setup/ticket/etc. If he's still around, pull all work from him and have him focus on documenting his setup.

Reply

[-]

SecurePackets@reddit

If there’s no documentation on how it works… time to grow up to a mature organization with documentation

Reply

[-]

edoceo@reddit

I've been through this more than once; like others have said...compensate this person for creating some documentation. Then second, find someone who can dig through all that and get them some face time with the person leaving (if possible). One time, we had the lead just abruptly quit and didn't want to chat at all. So, we had about three months of digging through piles of code. One part that was frustrating for everyone was that understanding the system required that both technical and business stakeholders would work together -- because we weren't just trying to find out **what** it did (the code shows that) -- but needed to know **why** it's doing things that way. That created a lot of tension.

Reply

[-]

Leucippus1@reddit

So, side note, if you are 'the guy' that does this - make sure you store everything in Git with markdown files, or have something like an ansible or rundeck where it is easy to discern what everything is actually doing.

Reply

[-]

spazzvogel@reddit

Brain dumps are important and if not, screen recordings are even better. Good luck

Reply

[-]

Livid_Switch302@reddit

Not trying to be harsh but this sounds like poor operational discipline. Any workflow critical to the business should be readable, documented, and understood by more than one person no matter how "good" he is was

Reply

[-]

sryan2k1@reddit

One thing at a time. Try to start figuring it out before it breaks. Also, don't let the org get into this situation.

Reply