Yelling at IT staff does not a business continuity plan make.
Posted by AudibleNod@reddit | talesfromtechsupport | View on Reddit | 104 comments
This is from a few years ago. I was working at a medium sized company as an IT sys admin. The company had just recently moved to a new location that was able to more comfortable accommodate its operation. It had an on-site call center as well as a medium-scale manufacturing/repair center. Since we were new tenants and everyone was now under one roof, many things were still being figured out.
One day, we got notice of a gas leak in the manufacturing area. We didn't have an alarm system for a gas leak so people were running around telling everyone else that there was a mandatory evacuation of the building. The IT people all had laptops so we all grabbed them and made our way to our cars. By coincidence our director of IT and the head of IT support were on a business trip. As I'm walking out the door the Call Center Director (I'll call him Cal) start yelling at me and the other Sys Admin. "Hey, what are you guys going to do!?"
"Go to our cars."
"No, no you can't. We can't receive calls. You have to do something!"
I turned to my coworker and we both realized that the call center still used desktop computers and soft phones. They couldn't do their job. Cal was red in the face trying to slowly let people out the door to the outside. It was then that the fire department arrived probably to clear out the building officially. So I asked Cal, "What's your plan if there's a fire? Just do that."
"What? No, you need to do something."
I shrugged. "We can't do anything. The phone system probably doesn't work off of VPN." I was guessing at that. "Just follow your plan if there's a fire."
"You guys never gave us a plan for a fire." Cal responded.
Because of course it's IT's job to develop a business continuity plan for the entire company. More people were streaming out. It was then I decided to ignore him and go to my car. I tried to call the Director of IT in the slim chance the airplane diverted or was delayed. No answer. I looked up in the company SharePoint site for a business continuity plan or fire plan or something. But only found stuff for IT, including our offsite backup servers and how to run IT operations from VPN. There was nothing about moving our softphones to/through VPN.
Cal knocked on my car window after everyone was out of the building. "Well?!?"
I explained that there was no business continuity plan in the SharePoint site and IT didn't have anything in place to shift the softphones to VPN. Plus we didn't have enough laptops to support even half the call center. Cal didn't like my answer and walked over to the CEO who was the fire department. I could see Cal pointing at me and yelling. Clearly we were losing business. And clearly it wasn't just IT's fault, it was mine and mine alone.
The fire department cleared us to go back in after about 45 minutes. Later that day I had two meetings with Cal and the COO scheduled. Since IT was missing both leadership positions to travel I was the scapegoat. The first meeting was cancelled and the second the CEO stepped in and cancelled it since it was really the job of the Director of IT and a lowly sys admin shouldn't be in these meetings.
Nothing bad happened to me when the IT Director returned. And the company hired a consultant to develop an actual business continuity plan for fires, weather and other events. Turned out, IT shouldn't have a button they could press in the event of a gas leak. For several months Cal scowled at me after that every time we passed in the hall.
TL;DR Call Center Director assumes that because his department uses computers, any problem becomes an IT problem.
lokis_construction@reddit
We have a business continuity plan. We transfer all calls to YOUR cell phone - you are the Call Center Director after all.
NotYourNanny@reddit
Where I work, anything that uses electricity, and anything that someone can't identify - whether is uses electricity or not - is an IT problem.
-Cthaeh@reddit
The worst part about it, is i can usually figure it out too. Don't put it on me to decide if I'm willing to help, surely you know IT isn't going to replace the fire systems batteries...
DripPanDan@reddit
That's not just where you work. That's every business.
I have people telling me that managing their own 3rd party, external website subscriptions that they use for their department exclusively is somehow an IT problem. No, man, I have no idea how to configure your business segment for your optimal workflow. That's really up to you.
I swear - for as much as people assume IT can do, they should fire everyone else and just have us run the company.
Geminii27@reddit
There really needs to be a formal scope-of-work statement done up for the IT department, saying what it is and is not responsible for, if only so it can be shown to people who try to dump things on IT.
The IT department is not responsible for:.
1) Anything not in the "IT is responsible for" scope statement
2) Anything which doesn't use electricity, unless specifically exempted.
3) Anything which doesn't use electronics, unless specifically exempted.
4) Anything which isn't specifically Information Technology.
5) Anything which is privately or personally owned.
6) Training people on how to do their jobs, even if that job involves using a computer. See the manager of that job, or the Training department.
7) Anything which was purchased, installed, or set up without the input of the IT department. If you want IT to support something new, set up a meeting with the IT manager, and we'll discuss what supporting that new thing will cost you. Bring your cost code, budget, and any agreements you have from Finance to cover the ongoing business/support costs of this item.
8) [Insert personal pet peeve here]
Warrlock608@reddit
This one drives me absolutely insane. If your job involves using a computer 75% of the day and you don't know how to use a computer, you are bad at your job. Full stop, you suck. It isn't my job to remedy that, it is yours.
Geminii27@reddit
It's not even 'using a computer', it's 'using employer-provided equipment with employer-provided job training'. There isn't a Desk Department to teach you how to use your desk, or a Phone Department to teach you how to use a deskphone. If the company provides a moderately complex bit of corporate equipment required for the job, that is job training; there aren't specialist departments training on each individual bit of supplied kit.
ToothlessGuitarMaker@reddit
I worked freelance support for a couple decades, home visits and consumer-grade hardware, and point 7 was proactively addressed by my policy: I will go shopping with you, for free, any time, to help you get the right thing, then offer a discount on setting it up, because doing it that way is easier and cheaper for both of us compared to you getting the wrong thing on your own then asking me to fix it.
spin81@reddit
This is not always true. Not trying to be pedantic here, I just want to point out that I'm happy not to be in a place where BYOD is a thing where you have to respect that it's people's own device and somehow still be responsible for it not getting infested with malware.
AlexisFR@reddit
Have you watched Silo? IT owns the world.
meitemark@reddit
Yes, we do. And we use it to display cat pictures and memes.
mercurygreen@reddit
I use a bucket metaphor:
If you request one, I.T. will provide you with a bucket. Our job is to make sure the bucket is sturdy, does not leak, and has a solid handle. You can fill it with water. You can fill it with rocks. You can fill it with sand and build sandcastles with it. If you break it, we will do our best to fix or replace it. But if you decide to use it as a hat, that's on you; we are NOT going to tell you how to play with your bucket.
spin81@reddit
I like this because it goes a long way. I'll be stealing this going forward.
Like if you melt your bucket because you decided to put it in the oven, we'll replace it but you need to realize going in that buckets cost money, and someone has to pay that money, and that someone may be you if you are not supposed to be baking upside down bucket pies.
My boss compares IT to water from a tap. Like you turn on the tap, water comes out. It just works. You don't care how or why it works, until it stops working. But it's not free for water to keep "just working". It takes maintenance and effort, irrespective of whether you care about it or not.
aussie_nub@reddit
To be fair, they should be managed by IT so that you can tell them when they shouldn't be using them.
AudibleNod@reddit (OP)
But not pay us any more when they don't notice any IT-related problems.
zorander6@reddit
But the coffee machine isn't working!!!!!!11111!!!!!!!
NotTheOnlyGamer@reddit
That could be a legitimate IT concern. I don't know anyone in IT around me who doesn't run on a steady diet of coffee and spite.
dhardyuk@reddit
If you last long enough in IT you have to switch to decaf before your kidneys need to be upgraded or just retire themselves.
When your kidney function halves for no obvious reason you’ll have buyers remorse.
The unfortunate thing is that kidney function doesn’t start from 100% by the time you are having measured. So your starting point might be 72% but supported by 2 kidneys. When that halves it’s now 36% which is liveable, not thrivable.
But that’s across 2 of them. If one properly gives up at that point you are now on 18%. Which is 1/5 of what it used to be.
That’s a shitty spot to find yourself in, it’s never drinking alcohol, or strong coffee. Or even mild tasting coffee that has stuff for your kidneys to deal with. You know, the good stuff, that you like getting from the coffee.
As someone who watched my kidney function drop in a matter of days from 70ish % to 35ish% I can tell you finding the balance of fluid to electrolytes needed to be kind to my kidneys wasn’t great.
Drink only water, risk low electrolytes. Drink a nice coffee, risk waking up with sore kidneys and the first slash of the day being a shade of brown a real ale would aspire to.
It’s horrid.
Mine has slowly climbed back up to the low 60ish % and I’m blaming a vitamin D supplement for it. I now take one vitamin D a fortnight, not one every day.
But years of driving 1000s of miles a month whilst also doing a full 8 hours of tech lead and support every day trained my bladder to hold the output of well over 4 pints of input before needing to offload. And that’s drinking all the stupid shit full of fizz and chemicals as well as coffee.
It’s not great, it’s completely self inflicted and entirely avoidable. Like the arthritis in my right knee caused by all that driving.
The only good thing about it all is that I’m senior enough to not do the driving and drink 1 caf coffee a day and everything else is decaf.
The stoopid shit you do when you are young and invincible catches up with you.
meitemark@reddit
Since IT run on coffee, fixing the coffee machine may sound like an IT problem, but it (the fixening) will go verly slow and with lots of errors due to lack of brain fuel (coffee) and in the end, the brown sludge that comes out of the machine is akin to a warcrime. It will however contain a weeks worth of caffeine in a cup.
The shaking of your hands and the rate of your heart is due to your mistake in asking addicts to fix your drug dispenser.
AudibleNod@reddit (OP)
I worked at a place that had "how to refill the coffee machine" as one of its knowledge base articles. Turned out the guy who wrote it did so in order to show all the steps needed to properly write a knowledge base article.
robophile-ta@reddit
Classic wikiHow
spin81@reddit
Huh that's a good idea.
Maxfire2008@reddit
Now they have a knowledge base article for "how to write a knowledge base article"?
sypie1@reddit
Filed under KB000001
Geminii27@reddit
KB000000 is presumably the article which details the existence of the knowledge base and contains an index to the major sections.
Maxfire2008@reddit
*KB000002
KB000001 is the one of which the second details the creation.
marysalad@reddit
In the beginning, there was the word...
mercurygreen@reddit
...and the work was COFFEE!
inucune@reddit
But the word was spoken, and could not be trusted. Man sought words unchanging, and endeavored to carve them into materials.
NotYourNanny@reddit
And if you make me work on it, I guarantee it will never, ever work again. (How can anything smell that good, and taste that bad?)
AudibleNod@reddit (OP)
That place was like that. Vending machines, parking lot lights and LP air hosing was all of IT's problem.
mercurygreen@reddit
I had someone try to tell me that toilets were our problem as well "because you have tools!"
sueelleker@reddit
When you have a hammer, every problem looks like a nail.
mercurygreen@reddit
Certainly ever *user* looks like a nail.
Techn0ght@reddit
And since the rest of IT doesn't know anything about the OSI model or connectivity, including plugging in the actual cable, it always fell to the Network team to figure things out.
NotYourNanny@reddit
Only thing I don't have to deal with is alarm systems, which my boss is very territorial about (being the top guy when it comes to security). Especially now that they all have their own built in cell phone, and don't need POTS lines any more.
Aselleus@reddit
I got a ticket for a store that had a literal fire because an old large capacitor blew, and they wanted me to check why there was no power to the computers in a certain section. The whole freaking section had no power. I'm like uh it's not a computer problem.
notfork@reddit
I heard it all, people with out power to building, people who's houses have literally burnt to the ground, people in the middle of hurricanes, and people who were superposed to be evacuated due to bomb threats. all calling in demanding their internet back up and running or someone out TODAY to fix it. No matter the issue that caused no working magic box, we are expected to get magic box working post haste.
Langager90@reddit
This kind of gave me an idea of a skit where a very, very weary-looking man in polo and slacks wanders by a police line, past negotiators, through the SWAT-issue barrier, and past a couple of armed gunmen to reset a router.
EruditeLegume@reddit
XKCD kinda got there first...
https://www.xkcd.com/705/
mercurygreen@reddit
Currently, in the Pacific Northwest, we're dealing with the aftermath of a cyclone. "Why can't I get access to the printers?" Because we have NO POWER!
spin81@reddit
You can only prepare for so much. I mean you can have like a PSU but are you going to say "you know a cyclone might hit so I'd better get a diesel aggregate I can use in the event of several days or weeks of power outages"? I think many if not most places wouldn't.
NotYourNanny@reddit
At least the three calls I've gotten about actual fires involved computers (one was a power strip, but the cash register was plugged into it).
canada432@reddit
This is where I work now. If it gets plugged in or has a battery, it's ITs responsibility. I've been called for everything from HVAC to bad outlets in a conference room.
NotYourNanny@reddit
I currently have four inflated beach balls in my office. They are not battery operated and do not plug in, but I guess they're IT beach balls.
Tactical_Fleshlite@reddit
I feel like stuff comes about because if you’re vaguely handy and volunteer one time, and you manage to fix it, it now belongs to your department.
To any young people reading, don’t volunteer to fix stuff at work lol, or at least outside the scope of your position. You will not be rewarded, you will just have a new expectation.
NotYourNanny@reddit
The punishment for a job well done is a harder job.
Geminii27@reddit
I've certainly had people try to make such things my problem when I was the only IT guy on a site. Or even when it didn't use electricity at all, but was made out of metal and performed a function.
We HAD an admin person for facilities stuff. But no, apparently anything that someone didn't know how to repair themselves was an IT issue.
meitemark@reddit
It it works and I don't understand how. It is magic. IT is pure magic. Therefore IT/Helldesk must be able to fix this thing. They have a magic "Fix it"-button. Admin person for facilities stuff takes way to much time, IT just have to push the magic button and it works.
davidgrayPhotography@reddit
One of my former bosses said one of the other IT guys (who doubles as an AV guy, doing projector related stuff etc.) was responsible for something that was decidedly not IT related (tidying up someone's cables at the back of their computer, I think), because it has "a plug on the other end"
So for the longest time, his nickname in our private group chat was John "Plug on the Other End" Smith, and every time something comes up that definitely isn't IT related, like sourcing a 4-way power board, we say "John, this is your responsibility. Remember, [former boss's name] said you're in charge of stuff if it has a plug on the other end"
drinianrose@reddit
I worked at a company once where people would call the IT "helpdesk" when they needed an office chair assembled.
harrywwc@reddit
well! you're there to 'help' - see‽ it's right there in the name!
Geminii27@reddit
"I need some Help with something that goes under my Desk"
joule_thief@reddit
Or when a toilet is clogged. Talk about a shitty ticket, am I right? /s
Solarwinds-123@reddit
Or when the blinds are stuck. Those are not the Windows I put on my resume!
SpongeJake@reddit
Oh yes. So true. Ever get a request for a USB fan? We did. And I think a USB heater for a user’s morning coffee. Or as you said, anything electrical at all.
“Hello, IT? What do we need to do to get a floor heater installed at various cubicles?”
“Well if I were you, I guess I’d pull out my credit card and hop onto Amazon.”
Geminii27@reddit
I mean, that's a Facilities issue. As is the subsequent supply of electricity in the area.
SpongeJake@reddit
Yes but those things have an on-off switch. Ergo it’s IT business.
ben_sphynx@reddit
To be fair, they could mess up IT equipment by just installing a floor heater and overloading the power said IT equipment uses.
Trinitykill@reddit
Had that. In a computer suite, a staff member had bought in a portable air conditioning unit, as well as several other appliances.
With all of them running, the breaker would trip any time more than 24 desktops were running.
They would constantly complain that they needed at least 32 desktops in that area, but every single time, I would clarify it's an electrical problem, not a technical one.
I'm not qualified to install higher capacity breakers, and if I was, it's not my job to.
TechGundam@reddit
For mine, add heavy lifting and furniture assembly.
Only1alive@reddit
Yup. Add "Home ISP" to the list and that sounds about right
mercurygreen@reddit
One of the WORST calls I ever had at an MSP was a guy trying to get me to fix his home ISP. The guy next to me called his boss just so he could let him listen in...
AntonOlsen@reddit
They rolled facilities into IT here, so we got it all.
NotYourNanny@reddit
At least they're not pretending.
KupoMcMog@reddit
We joke that when we swap the water jug, we should put in a ticket.
nighthawke75@reddit
When a client had a fire drill, I walked down the hallways sweeping each office for unlocked desktops(HIPAA, you know), Noting room numbers of who did not lock or logged off.
I gave their admin the nauty list and let her use the ruler on the back of their hands.
Pity, I looked forward to wielding that stainless beast.
StoicJim@reddit
There should be a backup site.
ecp001@reddit
Some of the coffee makers have digital clocks so they, also, are IT's responsibility.
bv915@reddit
Oof. May wanna look into that one. I'm willing to bet the fire marshal or OSHA would like to have a word.
zeus204013@reddit
Is like Cal harrassed you. HR may be helpful (with boss support).
AudibleNod@reddit (OP)
No. It was just a scowl. He was otherwise professional in all his emails and communication. He wasn't petty or vindictive. And at some point he stopped.
SrulDog@reddit
How, exactly? Scowling is harrasment?
harrywwc@reddit
"lack of planning on your part, mr. CCD, does not constitute an emergency on my (or even all of IT's) part."
AudibleNod@reddit (OP)
That's the long and the short of it.
nj_tech_guy@reddit
Dang what a champ of a CEO! Running a company and he's the entire fire department?!
JustSomeGuy_56@reddit
Every place I worked had some sort of business continuity plan. I only saw one that would actually work. They are expensive and require constant updates and drills and few companies are willing to spend the money. So they cook up some nice PowerPoint slides, and maybe pay for a backup site for the servers call it a day.
I do recall one company that was in the North Tower of the WTC. They had a complete backup facility with desks, telephones, computers etc. Unfortunately it was in the WTC South Tower.
Techn0ght@reddit
Reminds me of the diverse path for the network cabling that ran through a tunnel on the East Coast. It ran on the other side of the tunnel.
https://en.wikipedia.org/wiki/Howard_Street_Tunnel_fire
Tatermen@reddit
You'd be amazed at the amount of C-levels that think that ordering two circuits between A and B, is the same thing as ordering two diverse circuits between A and B. It's not, and the latter is usually much more expensive.
Techn0ght@reddit
Actually no, no I would not. It's hard enough to get them to believe redundant circuits are necessary let alone "Why aren't they the same price?" Don't get me started on diverse entrance facilities.
MikeSchwab63@reddit
Cantor Fitzgerald. A co worker saw some of them (with visible injuries) at IBM Boulder during an exercise in October.
KelemvorSparkyfox@reddit
A former employer once moved their production and backup servers from the then-hosting company's primary and secondary sites (different sides of the city) to the support company's primary and secondary data centres (two buildings in Nottingham).
This led to the then-Service Delivery Manager saying something actually accurate - "This means that when an aeroplane crashes in $Company's first data centre, the wing can bounce on down the road and take out the backup a few minutes later."
Exodus2791@reddit
Well, at least they tried. We once had a CRM secondary/ backup server in the same rack as the primary server. So when that particular data centre had a power outage....
TheCarbonthief@reddit
IMO the way to do it is just already have everything in the cloud, have work from home be something that's common and built into the culture. In 2020 when lockdowns started, my company just went home and we did the same exact thing we always do, just from home instead of from an office. There was nothing to rollover, nothing to migrate, nothing to configure. We just wrangled some laptops together, and when we ran out of laptops we sent desktops and monitors home with people. It was all AzureAD, literally just log in, the end.
Keep it simple, fight back against anyone that wants to add something that complicates your shit, and continuity can be a breeze.
Flat-Distance-2194@reddit
I’ve actually had to deal with idiots like Cal on a daily basis. One of the Datacentres I managed had a fire in the hall, gas suppression worked perfectly, out of an estimated 10,000 drives we lost 80. No bad considering the gas discharge blew the bottom out of a wall between the hall and storage. Turns out the wall was built on top of the suspended floor with no anchorage to the slab floor.
Guess who got blamed for that? Yep, if a server psu hadn’t failed spectacularly then the gas wouldn’t have been discharged ( 200 cylinders,80kg, of Argonite) . Never mind that the wall was installed two years before we took the building over.
Had to write a report that was basically, gas discharge because of server psu, wall failure- hire a consultant as I’ve no idea why it slipped and no idea on how to fix it. Not my problem. Gas can’t be refilled overnight as we don’t carry any in stock. On a good note, majority of the drives survived and those that were lost was backed up or backups so no data lost.
Still got asked what was going on with the wall daily until it was rebuilt, even though it had been shoved in to the hands of the Property Management team. They even put their own guy on site to liaise with IT and Facilities so we didn’t get in each others way.
We got six months overtime changing every single drive in hall one , couldn’t take the risk of a catastrophic failure after they’d all bounced during the gas discharge . Had a shredder in the car park shredding the drives into little pieces. After all you only need so many spare 120gb drives spare at home and everyone had enough. I think I only used the last one , last year, 10yrs after the incident.
meitemark@reddit
A button that disables the phone system, only outgoing calls to 911. A message to the callers that you are temporary closed, please call back later. A system that shuts down all computers and servers safely. The servers may do a last push of very important backups to offsite storage. Displaying "Please evacuate / gtfo" on all info screens.
A that kind of button? You should have. Other manglement may be less inclined to think of the safety of the workers.
me_groovy@reddit
"Hey, what are you guys going to do!?"
"I don't know, what does YOUR business continuity plan say to do about YOUR department, of which YOU are in charge?"
Remarkable_Job8691@reddit
Wow, what a frustrating situation! As someone who's worked extensively with business continuity planning, I can totally relate to the chaos you experienced. It's crazy how often companies overlook basic emergency protocols until it's too late. I've seen firsthand how a lack of clear plans can lead to finger-pointing and blame games.
Your story really highlights the importance of having a comprehensive strategy that covers all departments, not just IT. It's great that your company eventually brought in a consultant to develop a proper plan. In my experience, the best continuity strategies are collaborative efforts that evolve over time.
I actually created Venture Plus to help businesses avoid exactly these kinds of pitfalls. Having a solid plan in place before emergencies strike can make all the difference. Anyway, kudos to you for keeping your cool in a tough situation! Hopefully Cal eventually realized IT wasn't solely to blame.
DolanUser@reddit
And by all honesty… what did he expect to have as a plan in case of a fire/gas leak/other lethal event where you have to LEAVE RIGHT F NOW??? “Oh no, wait . I’ll just quickly run some scripts to switch over the call center.” Some people man…
katmndoo@reddit
It had some faults, but at least the call center I worked in had a really sane plan for evacuations. "Get out."
No one was taking calls at that point.
aussie_nub@reddit
Sounds like you should have put something into HR about you being harassed by Cal.
CharcoalGreyWolf@reddit
Every time he scowled at you, I’d have smiled sweetly as if someone just poured honey in my mouth.
mercurygreen@reddit
Considering that only four years ago we all went from in person to remote work in the space of a few weeks you'd THINK more places would have a working continuity plan now...
mercurygreen@reddit
I cannot stand people who think that SOMEHOW the Office of I.T will solve all their problems. (Yes, we're smarter than they are, but they can't pay us enough to care THAT much!)
virtueavatar@reddit
ok so here's what you should have done, once you were in the car and he tapped on your window.
Turn on the ignition, and then just drive forward a little bit, not far, just so he's not at your window anymore, then stop there.
When he walks over to your window again, just drive forward a little more. and just keep repeating that.
Use the situation to entertain yourself since you know it's their fault and their problem. Enjoy a bit of a giggle.
You can even just drive back to where you were once he gives up.
ThunderDwn@reddit
BCP is only an issue when something fails badly enough to need it.
Until then it's a distracting afterthought.
KelemvorSparkyfox@reddit
So a BCP is a backup solution, writ large.
Teknikal_Domain@reddit
But then the disaster is resolved and "we got through this just fine, there's no budget to develop a BCP after we've shown that we manage just fine without"
ThunderDwn@reddit
Do you work at my company in senior management?
Teknikal_Domain@reddit
No, I work for the railroads.
...... Yes I have a bottle of Jack in my hand.
dannybau87@reddit
Has nobody told you that everything is IT's fault?
If we don't have a crystal ball, a magic wand and everything works however a user decides it should work we have failed.
jezwel@reddit
COVID accelerated our BCP as anyone that wasn't front line customer facing was pushed to work from home.
Laptops users weren't a problem, but they were about 1/3 of the company - the CFO was too tight to increase that % in years prior.
We temporarily added RDP access so people could run their desktops from their own PCs at home (if they had them) plus added virtual infrastructure for cloud PCs.
The main point is that BCP costs money, and until the purse strings are opened there's only so much IT can do with what it has.
Now there's something like 85% laptops so BCP is a lot easier to manage.
logosolos@reddit
I definitely don't miss the private sector. Thanks for the reminder.
emax4@reddit
"Uh oh, Cal's here. I'm detecting a lot of hot air..." (keep hand inches from the gas alarm button)