Don’t make the business’s risk your own.
Posted by jkdjeff@reddit | sysadmin | View on Reddit | 63 comments
I see posts in here all the time (what prompted me to finally write this post was the one that popped up about a giant excel spreadsheet pretending to be an access review mechanism) where people talk about a process or practice that they can see is wrong, but that the business refuses to change.
When that happens? Give up.
You are there to give your expert opinion. Once you’ve done that? Your responsibility has ended. Let it go.
There are virtually no circumstances under which you would face any individual liability (ensure you are covered against those if they apply) and businesses make bad decisions all the time in a variety of arenas. Let them.
I get it, it’s frustrating to sit by while something is being done “wrong” but all you’re doing is stressing yourself out and potentially creating needless conflict.
Obviously, the higher up the food chain you go, the less this applies. This post is mainly aimed at individual contributors.
WonderDowntown3349@reddit
If you flag an unencrypted RDS instance sitting in production and three managers ignore it over two months, you just send a final email and drop it. When it blows up, you have the receipts.
The important thing is to document everything.
Leinheart@reddit
And then get fired anyway once shit gets ransomwared because right and wrong dont actually matter when it comes to rich people's theoretical money.
drashna@reddit
and then sue for wrongful termination, and retaliation?
Leinheart@reddit
You think I can afford better lawyers than the average corporation? No chance.
drashna@reddit
Ah, yes, because that's the only deciding factor.......
Leinheart@reddit
I dont generally file lawsuits i know I'm going to lose, when I dont have a job. You do you tho.
mnvoronin@reddit
Or, you know, live in a country with some actual employee protections.
Leinheart@reddit
Please, take me in. 🙏
jdptechnc@reddit
True, but at least you don't burn yourself out trying to fight a battle that you already lost.
hkusp45css@reddit
That too. If I'm going to lose a fight, I'd rather not lose after wearing myself out on a something inevitable.
hkusp45css@reddit
You can't control the actions of others. You can only make it so that you can provide evidence to rational actors and THEY can recognize that you did what you could.
ExceptionEX@reddit
I mean, not to say that doesn't happen, but in many instances right, the person fixing the problem, and the one that cautioned against it, isn't the one getting fired.
OregonTechHead@reddit
If someone is going to fire you for someone else's screw ups, do you really want to work there anyway?
BrainWaveCC@reddit
So, if it's going to happen anyway, why have unnecessary stress along the journey?
Warn them, document it, and move your focus onto something else.
Centimane@reddit
In those scenarios the only thing to have done differently was to find a new job before things go wrong.
Fighting it more wouldn't have made the difference.
jwalker107@reddit
I keep seeing this sentiment, but...has anyone actually been in a situation where having the email mattered? I've seen cases where an admin was let go for reasons good, and for reasons bad, but I've never actually seen a situation where they were targeted for dismissal but then saved by an email.
I'm not trying to be argumentative, this is a real ask - has anyone seen this work?
MissionSpecialist@reddit
The purpose of the email isn't to pull it out when you're already in the termination meeting with HR; it's to direct the eye of Sauron elsewhere when the very first question is asked.
I have done (or not done) some stupid things in my career when so directed by higher ups. When asked "Who approved this?" or "Who was aware of this?", I provided a clear answer with the relevant email attached.
In some instances, those named individuals left the company within hours to weeks. Would I have been the one leaving, absent that email? Impossible to say for sure, but having the email certainly moved the focus off of me entirely.
a60v@reddit
This. I've seen people waste everyone's time getting stuff in writing to validate bad decisions, but I've never seen or heard of it saving anyone's job. I see zero value to it. If there is a legal/compliance or safety issue where this might matter from a liability perspective, then the only right thing to do is to quit if it cannot be quickly resolved.
Synergythepariah@reddit
Better to have it and not need it.
Cheech47@reddit
I hate to pile on the agree bandwagon, but especially in an "employment at-will" state, if they want to fire you then cousin, you gonna be fired. Receipts showing it was your manager that made the call will more than likely poison the well between your (old) manager and their upstream chain of command, but if you're getting the axe and not him, then what good did it do?
trueppp@reddit
If you're in a place with labor laws, it can do a lot of good.
trueppp@reddit
The emails are not to save your job, they are there as leverage for your seversnce agreement and potential settlement depending on your local labor laws.
bv728@reddit
The problem is that for all people love writing fiction, it's very uncommon that folks can conclusively show it saved their job. I'm pretty sure it's helped save mine twice: I'd notified people of a specific issue, proposed a remediation path, was turned down, floated it up the chain, and left it be. Then the shit hit the fan and I got dragged into executive meetings, showed my receipts, and walked away unscathed. I've been promoted by that org since both of those.
Now, was this because I had written receipts? Was it because I have a history of getting the work done? Was it because I have good relationships with folks higher on the chain? Who knows, nobody's going to say it specifically. I suspect it's a mix. But I'm pretty sure they were looking to see if they were going to fire me, and the documentation was in my favor.
OregonTechHead@reddit
If it's at the point you're being terminated, it's unlikely due to a single incident. In that case, no, it's not going to "save" you.
However, if it's a single major high visibility incident where you're doing a moratorium, having that documentation that you raised an alarm before it happened is critical to protecting yourself, and preventing things like that from happening again.
It's also important to understand that "ignored" isn't always the case. Sometimes an issue is raised, discussed with other leadership, and determined no action was needed. Poor management might not communicate that, and the end result appears to be it fell on deaf ears.
notarealaccount223@reddit
I'll add that you want to spell out the risk (i.e.whats the worst that can happen), the effort and cost to remediate, the user facing changes and explicitly ask for a decision on moving forward.
That provides a smoking gun if something goes sideways.
In the report after you can state, "Management was notified of this risk on xx, you & zz dates and chose to accept the risk." Remember it's blameless so don't name names. Good leadership will ask after this occurs (sometimes it takes a few incidents explained with similar wording)
There should be a section that also noted what can be done to prevent future occursnces. It should mirror the email sent to management.
TheRealLazloFalconi@reddit
THIS! You can't just say "Hey this is a vulnerability we need to fix." You need to let managers know what the actual consequences are, and importantly how much it will cost to fix before vs after it blows up.
CeC-P@reddit
That's what pissed everyone off at my last 4 jobs. I'd always be doing things that "weren't technically my job" that weren't anyone's job or weren't being done by anyone else. Like if I'm level 2-3 tech support, I'm crafting the entire fix because our engineering department sucks. Someone put in a ticket and I solved it. Get over your ego and put it on the KB.
Another company simply wouldn't listen to me or stay out of my way and constantly got hacked, had outages, had downtime, etc. I put my notes on the post-incident report, threw everyone under the bus but myself, then quit because the entire company was run by idiots and almost went bankrupt 3 times. Mostly after merging with a company full of literal crimnals with pending lawsuits against them.
IT IS NOT WORTH IT! It is not worth the stress. Companies like that don't need to be propped up by my effort and skill level if I'm the only one trying.
MaelstromFL@reddit
Story time!
In 2001 I was running a team that installed enterprise software. The company developed it, we installed and upgraded it for clients. Made a deal that we would own the install of the QA environment, this worked, because my guys would get a first look at the new upgrades and installs before we would hit the road to the clients.
I September, Nimda hit! All the SQL Servers in the QA environments became super spreader. My team bucleled down and assisted the developers in creating the cleaning CD used to fix our environments company wide. (Out developers actually worked out a few of the fixes that were used world wide!)
Two weeks after everything calmed down, I was summoned to the CTO. I had 2 days to compile my information and get to the HQ in Columbia, SC and, "explain why the servers in my control were not properly patched and contributed to the virus spread."
I walked into the office about half an hour early to my time, and went to the Admin Assistant to the CTO. She looked at me with fear in her puffy eyes and said," Oh no, not you too! He has already fired 3 people this morning!" I had fostered a friendship with her, always good to get a quick email from her about new changes before they were announced. I told her that I thought I would be fine.
I get called into the office, sit down in front of his desk and he tells me that I had 7 servers that were infected and not properly patched. And asked me to explain why...
I reach into my bag and pull out a thick folder of about 100 pages and place it on his desk. I tell him that it is all of the emails to the developers telling them that I cannot patch the servers because they used SQL Injection in their code and the patch would break the software, please fix. I pull out another folder, about half the size of the first, explaining that these are my escalations to department heads and directors on this issue. Finally, I pull out a small folder, and explain that these were the 3 emails directly to him asking for assistance in fixing this issue.
He only opened the small folder, browsed over it for a bit, and thanked me for my time before telling me to leave. By the way, the director of development and VP of engineering were both fired.
Cover Your Ass, boys and girls!
DaftPump@reddit
Yet he was not fired? He's the captain of the tech ship and seems he failed to keep watch.
BerkeleyFarmGirl@reddit
This is a great story.
The other day I was telling my co-workers that I was part of an org with MS Premium Support at the time and was on a very long call after Nimda/Slammer where there were multiple MS VPs saying "we're so sorry".
MaelstromFL@reddit
I have told it before, there is a lot cut out of it, but it really happened. I have learned that you can't force compliance from the IT role, you advise, and then CYA. Because, it will come back at you!
TheGenericUser0815@reddit
Your story made my day! Thank you so much!
xixi2@reddit
I don't follow. The devs used SQL injection? Or their code was vulnerable to sql injection? How does that affect your ability to apply updates to the servers?
MaelstromFL@reddit
Yep, the developers use SQL Injection in their code. If I remember, SQL Server 2k closed it in Service Pack 2. Apply Service Pack, code fails...
I had been screaming about it for probably 6 months when Nimda hit.
pdp10@reddit
SQL injection happens in the web-logic layer, normally. Your service pack might have had something to do with prepared statements, though.
xixi2@reddit
How would patching a server have anything to do with the fact they wrote insecure software running on said server?
pdp10@reddit
A frequent rhetorical question here, is why have printers at all? This is why printers, even a quarter-century later.
MaelstromFL@reddit
It was banking and insurance, in 2k we still did a shit ton of paper.
Secret_Account07@reddit
Totally agree.
I work for a large org with several layers of mgmt and hundreds of techs. We still have 15 2012R2 servers that I cannot get mgmt to force the customer up upgrade. Several of them have 95/100 vulnerability scores. It’s a losing battle.
Our job is to provide advice. Managements job is to decide what to do with that. If you argue and try to circumvent the chain of command all you’re doing is risking your job. Gotta let mgmt make their own mistakes. Thats literally their job
TheGenericUser0815@reddit
I don't care for the chain of command when the fall of the empire is at hand. It's also about my job and income at the end of the day.
Curious201@reddit
this is a good reminder that being right about the risk does not mean you own the risk. i have seen admins get stuck in this loop where they document a weak backup process, shared accounts, no mfa, bad patching, or an ancient server, then keep mentally carrying it because management keeps ignoring the recommendation. at some point the cleanest thing you can do is put the risk in writing with the likely impact, give a realistic remediation path, and ask for an accept/remediate decision. after that, keep the receipts and stop turning every ignored business decision into your personal emergency. the tricky part is making sure your documentation is calm and specific enough that it protects you later instead of sounding like an emotional complaint.
TheGenericUser0815@reddit
When I se a risk I can't fix myself, because it needs money and a descision, I write a note and send it to a carefully selected group of people, so that nobody can say they didn't know. In German you'd call me "Eckenpisser" which would translate to corner pisser.
dinominant@reddit
There is personal exposure if your org is compromised in a way that impacts your future employment. If they are forced to close or shrink the workforce from very expensive damages, then that does impact you personally.
Just something to keep in mind.
tobascodagama@reddit
Document, document, document. And make sure you can access that documentation (or backups of it) even if your access to company resources gets cut. (Be careful about violating data collection/security policies, but a simple journal that says "I was asked about X, my suggestion was to do Y, the company chose to do Z" should be 100% safe to keep off-site in all but the most high-security environments.)
pdp10@reddit
Concerns are rarely about that kind of liability. In the workplace, concerns are usually around someone else's poor planning causing an emergency on your part.
Like the lack of useful redundancy due to budget, could result in routine off-hours emergencies. Then, to add insult to injury, a meeting where stakeholders demand that something be done.
Or a bad vendor decision, combined with a middle-management directive to "figure it out". Nothing is more frustrating than when an authority loudly demands "one throat to choke" just like the salespersons suggest, then goes missing when it unfortunately comes time to do some choking.
What we hate are "emergency projects". Avoidable emergency projects.
ninjaluvr@reddit
It's only frustrating because you let be. Why do you care if a project is an emergency or avoidable? Do the job, get paid. If you want to make a difference, move out of being a sysadmin and into leadership. Otherwise, do the job and get paid. Once you change your thinking and start enjoying the work, and I mean enjoying all the work, even the avoidable emergency projects, your life will be so much less stressful.
pdp10@reddit
Because it makes for high, unbudgeted costs and a rocky business relationship.
Your compensation negotiation is your bid on the job as you understand it. If someone decided to take a lower comp because of assurances that the job is 9 to 5, but then they find out that emergency off-hours work is routine, then the result is high, unbudgeted costs and a rocky business relationship.
I once volunteered to join the technical leads of a company we acquired, in a cage migration at an outside datacenter of theirs. Turned out that they liked to do these things all at once, in a 24-hour thrash, with no sleep or structured breaks. Also no checkpoints or fallback plans, or at least none that they would admit.
It turns out that I did not enjoy being in a datacenter for 24 straight hours. That was perhaps the epitome of avoidable, "emergency" projects. In terms of planning and responsibility to staff, their professionalism was low.
OregonTechHead@reddit
But that's not your problem, and (I believe anyway) the entire point of this thread.
Your job is to do whatever needs to be done. It's someone else's job to figure out how to pay for it.
With who? This wasn't your doing, and you're just there to help. If anything, these are the things that create great relationships as you're seen as someone saving/fixing a major issue.
What? Unless these emergencies aren't tech related, it's still part of your job.
Did you learn to not volunteer without fully understanding what you're volunteering for? Is that the best process? Probably not, and certainly not the way I would do it, but it was their process, and you should've understood that.
That's not an emergency project at all. That's just a project that you disagreed with on how to accomplish the task. There was no emergency about it.
pdp10@reddit
It appears that I've confused the matter by using the language of business, for individual professionals (who are in the business of being paid).
I'm talking about risk and cost transfer down to Individual Contributors. Off-books technical debt will tend to transfer risk and costs to the engineers and techs who carry that debt.
OP is suggesting not to electively take on technical debt, and I'm suggesting that technical debt often makes its own^* decision when it needs to be paid against.
-
-
"Don't anthropomorphize the technical debt," he says.
OregonTechHead@reddit
That risk isn't on an individual. It's corporate risk. And that tech debt also isn't on an individual. It's corporate debt, and the company made the decision to keep it.
You don't need to make it your own, which again, is the entire point of this thread.
Your condescending tone never ceases to amaze me.
pdp10@reddit
If someone is being woken in the middle of the night for an emergency that they reasonably didn't anticipate, that counts as a small cumulative example of risk transfer, don't you think?
ninjaluvr@reddit
As a sysadmin, being told to deliver a solution by your leadership with high costs isn't your concern. They've made the budgeting decision.
Sure, but that's a different problem. Your work schedule and compensation is your issue that you need to address with management. You can have avoidable emergency projects and still work 40 hours a week.
What you're describing is their operational model, not an emergency project. If you don't like their operating model, move on.
OregonTechHead@reddit
Who cares? Other than having to stop what you're working on, and others fluttering around in a frenzy, it's just another project.
If the emergency wasn't your doing, just work on it the same as any other thing.
Low-Okra7931@reddit
People act like they will get a share of the profit if they fix these problems, or anything really lol. It's comical to me.
altodor@reddit
Just to play devils advocate here, American's cut off the profit pays rent and health insurance. Right now jobs in tech appear to be in a slump. If the business goes under because it's doing things poorly, we can't pay my rent with "I told you so" emails.
notarealaccount223@reddit
Counter point. If it costs more to properly fix than the business can support you end up in the same position.
Which is why cost to remediate and potential impact to the business are important pieces of information to share in these messages.
All businesses have to accept risks. We need to help the business understand whoch IT related risks should not be accepted.
altodor@reddit
I feel like a lot of the time it's things the business can afford to do, but chooses not to.
Like I don't think most people are out here seriously recommending that their 200-person company have four backup ISPs even though that would basically guarantee them against downtime. It's quite often things like "hey everybody should be using MFA" or "maybe we don't need everybody in the company to have domain admin" or similar no-nonsense best practice things of that caliber.
notarealaccount223@reddit
Yeah most of these are common sense and should be done.
The take away is that IT does not exist on a vacuum.
_haha_oh_wow_@reddit
Point out the issues in writing to cover your ass. If they choose to ignore their subject matter experts, that's on them. When shit hits the fan, you can point back to your written warnings cautioning against the decision.
Expensive_Finger_973@reddit
This was a lesson I learned after many years of beating my head against that wall. I came into my current gig about 7 years ago with the explicit mindset that I was going to be an individual contributor and that is it.
I do my job, go a little above and beyond from time to time, prod the boss for a promotion occasionally, tell the boss if I think something is or is not a good idea (but always make it clear I will fall in line with the vision no matter what that is), then I go home.
At times it is still a struggle to let something fail that I know is going to fail because I have seen it fail at this place before and sometimes I break my own rule of not getting involved. And every time I do the thing still fails and I end up regretting associating my reputation with the failure by trying to save it.
I tell anyone that asks what my "philosophy" is when it comes to work that it is that the businesses problems are their problems, they just pay me as little as they think they can to try and solve those problems. But at the end of the day it is still their problems, not mine. My only problems are making sure I am as marketable as I can and having as much of a financial buffer for layoffs as I think is reasonable.
Vodor1@reddit
Well it's nice to see someone say something other than "if the business doesn't do what you suggest, it's time to get a new job!"
I'm sure there are plenty here that get that, but it seems so many posts are met with this strange passive hostility rather than actual advice.
shimoheihei2@reddit
The main problem I see is people taking their job as their own personal project. They get too invested. This isn't to say not being proud of what you do, but you don't own any of the systems you set up, don't be overly protective. You are exchanging your time for money, that's it.
Familiar-Yam-4200@reddit
They’re right, but it’s kinda hard to actually do. Seeing something wrong and just letting it go doesn’t sit right. But trying to fix everything all the time is exhausting too.