Succession planning in IT
Posted by antons83@reddit | sysadmin | View on Reddit | 47 comments
Hello everyone. Some quick background before the meat of the story. I have 18 years in one company - 12k endpoints. Worked my way up from helpdesk to sys admin. (12 yrs level 1, 4 years level 2 and 3, and then sys admin for the last 2 years.
I took over as sysadmin after we had a round of retirement packages. Our previous sysadmin had 20 years in this job. Between the time the package offer was handed to him, to the time he signed to when he left was about 6 months. It was terribly handled. He scrambled to write as much down and even offered to help me after he left. Good guy.
I am eligible to retire in 12 yrs. I don't have a Jr I can pass knowledge down to. Sure I can write things down, but it won't be the same as actual experience with hands-on training.
My question: Has anyone here had this happen, and how did you deal with it? Is there a path to sysadmin in your org? At what point should I start pushing management to hire a Jr, so the transition is smooth.
InstructionDirect773@reddit
18 years at one place is wild, and that climb from helpdesk all the way to sysadmin shows some serious dedication. so you're basically the institutional knowledge at this point for a 12k endpoint environment, which honestly sounds both valuable and like a lot of pressure. when you say you took over after retirements, are you the only sysadmin handling that whole infrastructure or do you have a small team under you now?
antons83@reddit (OP)
Oh definitely not the whole infra. I deal with security and policies. We are a team of three. One deals with servers and windows updates, and the other does VM, and together they deal with our images. I'm the sole admin for AD, Intune and defender policies. Also deal with most issues the L3 tech doesn't know. Yes, this institutional knowledge is what I'm worried about being lost. Because I have groundlevel knowledge, I feel some duty to pass it on to the next person. Sure, someone brand new can be taught my current job, but it's these nuances I've picked up over the last two decades that will be lost if they don't have some institutional knowledge. That will take some time. Our users base is also ranged. From physical labor, to purely office staff. Each have different requirements, and our management (unfortunately) likes to satisfy a lot of them. This means many unique fixes/Band-Aid solutions and policies implemented for small departments. So unless you have some understanding of what that specific team does, or what they're called, or who their official (and unofficial) contact people are, it gets difficult. Again, not the end of the world, but surely a 10 min fix turns into a two week fix, simply because a particular department has a particular set of policies or special fix applied.
rubber_galaxy@reddit
Are you planning on retiring in 12 years? If so it might be a bit premature to worry about hiring for a Jr admin especially if you can handle all the work now. Just I would suggest just keep writing things down, document everything and passwords. Maybe in 11 years time if you are still in work you can push to hire a successor but wouldn't worry about it now. Also not really your problem if you have done the most that you can.
STUNTPENlS@reddit
At the end of the day it isn't his issue to worry about. If his higher ups have their heads up their asses that's their problem.
"Not my monkeys not my circus."
waxwayne@reddit
What if he gets hit by a bus?
Vectan@reddit
How about wins the lotto?
waxwayne@reddit
I like that scenario better
graph_worlok@reddit
Everybody does - I’d hate to think of the factors that would result in people hoping you get hit by a bus rather than win the lottery!
Churn@reddit
I disagree unless the plan is to never be sick or take time off for the next 11 years. Hiring a Jr admin is not just about retirement. OP is a single point of failure.
Indecisive-one@reddit
OPs question was specifically about retirement and succession planning.
OwlsAudioExperience@reddit
Churns point is that in the event OP is forced to retire before those 12 years due to anything including illness (heaven forbid), he’s a single point of failure in the org.
Best to get a junior or backup sooner than later.
TheDevauto@reddit
In 12 years it wont matter. However the key is documentation. Use an AI agent to keep daily journals of what you do, with detailed steps to triublshoot and solve issues and then compile that into docs in MD format. Just do it on the side. Later you can make it a wiki or something but thats what I would do in your place, even if I think in the next 10 years there will be few sysadmin type jobs.
sambodia85@reddit
I was part of a transition team that helped merge 7 Silo’d IT departments into 1. Most of the old teams made redundant. So many of them had the attitude that “nobody can do what I do”, “this company would fall over in a week without me”.
And they were right, a lot of stuff did fall over in a week without them, but this funny thing happened, we just carried on fixing it. And most of their special solutions that only they could come up with, either were done pretty much the same way in the other 6 silos, or if it was different, usually the unique one was worse.
So yeah, do whatever you think feels right morally and ethically, like leaving some breakglass accounts in a vault for the CEO. If you want to really help the nextadmin, document it in some kind of opinionated system like Hudu. If you want to get really pedantic you could map out the entire process of year to year running, system owners etc in some kind of vCIO software.
But 6 years on from the transition project, the only thing I find I consistently refer to from the old guys is a shiittt spreadsheet with all their vendors, with key contact info and logins to their various support portals,
Tsurting@reddit
Well, let’s make this an exercise. If tomorrow I were to join your company and was expected to take care of the infrastructure, this is what I would like to see:
From the network perspective: - How many sites/locations do you have? - Where physically are they located? - Do they connect to each other? If so, how, e.g. site-to-site VPN, MPLS, dedicated fibre connections, etc - For each site, an idea of what is there, stuff like DHCP/DNS servers, domain controllers, RADIUS (or other auth) servers, etc - What does the network look like? Are you using any routing protocols? If yes, a high level overview of the configuration/architecture - Do you have any copper/fiber patch panels anywhere? If yes diagrams are usually very appreciated to keep a record of how things are patched through
From a server-side perspective: - What do use for patching and updating software/applications/configs etc - Do you run anything on infrastructure hosted by a cloud provider, e.g. AWS, Google Cloud, Azure, etc - If you self-host, where is spare hardware located? Do you know how much of each spare component you have?
From a user perspective: - How do your users access and use your infrastructure? - Access files on NAS solution? - Use self-hosted specialised hardware to perform simulations, 3D renderings, etc - Share files directly with each other P2P - Do you have remote users? (VPN connection) - How are end user devices managed? (MDM)
You can go in way more details depending on what your environment looks like. But these would be my initial questions.
mad-ghost1@reddit
Doesn’t the MSP have a documentation?
So_average@reddit
I find it incredible that sysadmins aren't documenting everything as they work. We had notes about how to solve stuff stored on our OS/390 26 years ago.
Now with tools like Confluence and Jira and trouble ticket systems, you can create extremely useful knowledge bases with great search functionality.
Even text files on your pc would help future onboarding.
Shwabby89@reddit
Wtf you’re probably not going not going to be working for this company in 12 years just do what your told and nothing more. Why stress about it
hwsales@reddit
Let the people handing out the packages figure it out. If they are not adding people to grow within the company, a Junior, that is their problem.
RabidBlackSquirrel@reddit
CYA first though. Bring the single point of failure to their attention, outline the risks, and propose solutions. If they ignore it, that's on them.
Lordnerble@reddit
And when they crawl back begging for your help, Make them pay, You no longer need them, They need you.
Frothyleet@reddit
Yup, they have made the business decision that they are not worried about the bus factor.
Nathanielsan@reddit
12 years level 1? Seems insane.
antons83@reddit (OP)
Haha you're telling me! We're in a unionized org, so nobody really leaves. The upside is, we got paid a lot of money to take calls. We had guys with even more years in level 1. The good thing is, all of us had very detailed knowledge of the org. Policy settings, software peculiarities and the techs to contact and the question to ask them. This made us incredibly knowledgeable and our FCR was high 90s. Then we got bumped up to L2, and L3, and the level 1 jobs got outsourced overseas. The FCR dropped down to under 50%, and people were upset. But the company saved a ton of money so.. yeah.
Empty-Lingonberry133@reddit
Its seeming like this is becoming a bigger problem the more I look around and it's not isolated to your company. With AI rising and entry level jobs being removed the work force with high knowledge of an environment is aging out with no one to take up the mantle.. idiocracy the movie is.coming true but forced by AI
antons83@reddit (OP)
This is exactly what I'm talking about. Everyone keeps mentioning documentation, but documentation doesn't cover the knowledge a sys admin, or a master mechanic or electrician has for their particular org or product.
awful_at_internet@reddit
I am in Higher Ed. Our org had a major round of resignations/retirements about 6-8 years ago, which absolutely gutted our institutional knowledge; nothing was documented, maintenance had been deferred, and things were all around a mess. Not just IT; the whole org.
I started about 4 years ago as a student-worker, and have been full-time for about a year. A huge part of my role, in both positions, has been to hunt down broken links and outdated documentation, find its owner, and notify them that it needs updating... or write a replacement.
Documentation helps more than you think. It's also handy for yourself: how did you solve X problem when it came up 3 years ago?
BBO1007@reddit
Unless you have a specific retirement date that’s within the next year, I wouldn’t even bring this up.
12 years? Your company will most likely look vastly different.
Spong_Durnflungle@reddit
Let them bring you back at four times your hourly rate as a consultant on an as-needed basis.
That's what I did.
antons83@reddit (OP)
Haha they actually tried that with the sys admin I took over for. He refused, but he did give me his number to reach out to. I did call him often for the first 6 months
Anthropic_Principles@reddit
12 yrs? You're optimistic that there will be a functioning society in 12 yrs.
ArieHein@reddit
Sorry for the length, its just a topic dear to me :)
In one of my previous employers, we knew i was leaving after 3 years as i was moving countries.
For half a year, prior, i left 40 hours of videos from team meetings where we went over basics, then specifics to the org, to future targets, for each and every product we managed. I had a similar one with my managers including a summary presentationof directions.
With that employer, I had the opportunity to also train a young team for over a year on the how, so the videos served as a reminder but also on the why, which isnt always understood at the start.
In other employers, i always pushed to train the help desk, lower support ranks on basics because when they got better, it made my work better so i didnt wait to the end to the rush training. Mind you this requires mgmt buyin but even if not, go to some meerups, share, see if there are courses for young people in the uni where you can volunteer. Write a blog, internally and public.
I have had times with atocious handover to me, making me spend 2 months after a person left, running around caring for invoices as no one else was able or willing. Probably worst 2 months in my long career.
So treat you situation as if you have colleagues. Talk to the young people doing the job you did back then and be more proactive in their training.
My moto has always been around the fishing pole. But its more than that.
You entire career is split into 3: Third - youre a sponge. You learn anything and everything you can, or others are willing to share. Third youre the sysadmin. Youre solid in knowledge, start learning other areas that have some dependencies, even if not direct ownership, because we dont live in a silo. This is where you also make the decision - stay technical or go mgmt. If you stay technical, the third part is where the last choice is made: Continue being a master of your trade. Become a teacher/trainer. Knowledge sharing is fundamental core value in me. No matter what you think of your current job or managers, we all want to leave the place in better state than we got, at least for the one that will follow.
Im at a similar career expetency as you and i dont think i will ever stop. But this is a mindset. I understand not everyone is the same or has same reality. Im just paying forward for the opportunities i was fiven at the start to help others.
antons83@reddit (OP)
Thanks man. This all makes sense. I don't know if I mentioned this in another reply or not, but I in a very weird situation. We're a unionized org, and I'm one of the most junior with 17 years in. And on top of that, our level 1 is outsourced overseas. Our level 2 is ready to retire in the next 5 years, so they'll be gone in less time than me, and level 3 tech will be gone in 2 years. My goal is to find someone when/if my org hires a younger tech into lvl2, and train them up.
NotMedicine420@reddit
Take it with you. The ship goes down with the captain. They think the CEO is the captain. Fools! That will teach em.
llDemonll@reddit
It’s document things you think are important. If they don’t have someone you can work with for a while before you leave that’s their issue.
pdp10@reddit
Then the organization may need to hire a senior instead of a junior.
Your secrets vault and documentation should be written for a senior -- its all "tribal knowledge" that is not contained in normal outside documentation. Your docs should not be duplicating upstream documentation, in any way.
BoysenberryDue3637@reddit
Juuuust retired so understand this.
This is not a you problem, this is a they problem. If they have not done proper succession planning then it is on them. You could drop dead tomorrow with a heart attack or get hit by a bus.
I started planning for my retirement years before it actually happened. People above my grade pushed for training/plan on how to continue without me. I did this with all my people.
czenst@reddit
Somehow you managed and company kept on going - next guy will figure it out on his own as you did.
antons83@reddit (OP)
Yep, this is what I think about often as well. But IMO, I was only relatively successful because I had very specific knowledge of our org due to the time spent. This might be shocking, but they have not hired a fulltime IT tech after me. They outsourced level 1 to an overseas MSP, so those guys aren't being bumped up. The level 2 guys are under seven years before retirement and the one level 3 tech is two years away from retirement. Whoever they hire will have to be hired into level 2, after they lift the hiring freeze.
peter888chan@reddit
As others said, document things. If someone complains that you’re not doing sysadmin stuff, give them side eye. Other thing to do during those years, try to get rid of as much non standard stuff as possible.
Yuuku_S13@reddit
Sounds like you have good candidates at level 3, no? Train the level 3s how to do your job, not all of it… just enough to know what to do if SHTF and reduce the chaos in an emergency situation.
antons83@reddit (OP)
Our level 3 is one person. The reason I actually posted about this because of our particular (funny) circumstances. We have about 20 level 1 agents, but they're overseas. MSP. We have some about 8 level 2 techs, who are all under 7 years to retirement. There's me as the sysadmin for endpoints and I am the youngest one amongst everyone else. So technically whoever they move up will be eligible to retire in about half the time 😁. We're in a unionized environment, so they move people internally first.
Velvet_Samurai@reddit
I'm in the same boat I'm not super worried about it. My predecessor died unexpectedly. The stuff I couldn't figure out I replaced. Everything else was good to go. If someone can't come to grips with a system it can be entirely replaced. I can't think of anything that is sacrosanct to our business.
ledow@reddit
Document your work, as everyone should do.
Any qualified competent professional should be able pick up your documentation and find out everything they need to do ANYTHING specific to your employer (if not... your documentation sucks). P.S. You're not supposed to write "IT For Dummies" either, your documentation is aimed at PEOPLE LIKE YOU, not training up the next generation from scratch.
Beyond that - it's a problem for your employer.
ExceptionEX@reddit
The truth is, just document the best you can, in 12 years no one will be looking to maintain what you've put in place, it will likely be considered legacy, and the foundation of what they will replace. So having solid documentation and a little "why the sausage is made this way" will go a long way for those who come next.
I've found that practical documentation of infra is generally a lot more useful than a infra bible, probably labeled cables, clean installs, that you don't let turn into a rats nest over time, good color coding, consistent layout and behavior when possible.
Treat your infra as if you might get hit by a bus tomorrow and someone is walking in blind to replace you.
CraigAT@reddit
Writing documentation is a great start, but keeping it up to date and removing stuff that is no longer needed is even more useful - as we have all had to cope with old info or wade through useless info to find what we needed.
Set aside some time every month to add, review or update your documentation. Try to make it as "user preference" agnostic - your favourite note tool may not be another user's or worse may become a chargeable app.
If you had a Jr then I would say get them to walk through any of your documented processes, sometimes we assume knowledge that is not obvious to others.
If the business is good to you, give them a 6 or 3 months heads up to bring in a Jr or bring an MSP onboard whilst you are there. If they choose to not replace you in any way, then your documentation would need to be targeted at a much lower level.
FelisCantabrigiensis@reddit
Your organisation should be concerned about what happens if you fall ill or resign for some reason, let alone when you choose to retire. Either others should be able to do the necessary tasks, or there should be a way for someone new to take over easily (such as solid documentation and rational design).
You can help them with this, but if you're The One Sysadmin and the organisation doesn't have succession or continuity plans, then that's their choice.
Humble-Plankton2217@reddit
12 years from now today's knowledge will be less valuable. Things are changing quickly. Wait until you're about a year away from the finish line before you worry about this.
Also, people have different ways of doing things - some ways might be better than yours. All anyone coming behind you should need is a list of what systems are where and the credentials. Hopefully this is already documented somewhere.
Give me a list of servers, their functions/locations and the cred keeper app, and I'm ready to roll. Everything else is figure-outable.