HR & Fire Detectors

Posted by critchthegeek@reddit | talesfromtechsupport | View on Reddit | 54 comments

Same company as previous story.. the IT department (actually they called it MIS way back then) was on the lower/ground floor. The floor plan was offices, hallway, my office with glass wall, IT bullpen (my guys), another glass wall, computer room, another glass wall, hallway, more offices. So from my desk, I could look all the way through to the other side of the building. You could get into the computer room from either end if you had a card to swipe at the door. Nobody other than IT had those cards...

.....or so I thought...

Sitting there midmorning one day, pounding away on my keyboard and some movement caught my eye. Looking through my window, across the bullpen and through the computer room, I see the {expiative deleted} HR manager and some guy carrying what looks like a leaf blower (????). I'm rather P.O'd the HR had a card I didn't know about and just walked in there. They were looking at the ceiling and the guy raised the "leaf blower" and

OH CRAP!!!! That's a smoke wand and the idjits are "checking" the detectors

I vaulted over my desk, ran through the bull pen and into computer room just in time hear a IBM4361 mainframe, AS400 B50, Sparc fileserver, Novell fileserver, ROLM phone switch and (3) T1 muxes (for data/voice to the remote plants) all winding down to dead silence.

We didn't have a Halon system in there, thank the powers, but the smoke detectors killed the big UPS and all power in the room...

The HR guy and the other just stood there, eyes wide, mouths open with the patented "What just happened?" look.

And, with the glass walls, a bunch of other department managers, who came to see what happened, stood there and greatly enjoyed watch me jump up and down, ranting and raving at those two...

[-]

KelemvorSparkyfox@reddit

The first company I worked for that used an AS/400, it lived in a room behind a door with a magnetic lock. Said lock was set to fail-close. This was a Problem.

One day, some workmen outside the building put a JCB bucket through a power line. The office went dark and quiet. Then the screaming started. The comms room didn't have a UPS. The AS/400 had a backup battery, good for about half an hour. Cue much frantic discussion among people about how to open a powered door with no power. There was a solution, as it happened. >!They sent the skinniest IT guy up through the ceiling tiles (suspended ceiling) and over the wall that way. Once inside, he could perform a safe shut down.!<

The last company that I worked for that used one, it was hosted by an external support company, several counties away...

[-]

amishbill@reddit

Ouch.

Mine was similar - the door control system locked up one day. The only way in had a mag lock. The panel we needed to reset the mag lock controller was in the server room… behind a mag locked door.

[-]

KelemvorSparkyfox@reddit

"Please pry this box open. If you do not have a crowbar, there's one in the box."

[-]

Alpvax@reddit

I remember years ago (pre-everyone having access to emails on their phones) setting up a new router with my father. Written either on the bottom of the router or in the documentation was a notice stating "for help setting up your new router, please email xxx".

I remember seeing the irony in it even as a young child.

[-]

himitsumono@reddit

Or in the early days of Netscape; all the "help" files were on the internet. If you were connected to the internet, you didn't need them. If you were having trouble connecting to the internet, well you just kept having trouble.

Now everybody's emulated them. "Offline" has a new synonym. "Screwed."

[-]

apnorton@reddit

They sent the skinniest IT guy up through the ceiling tiles (suspended ceiling) and over the wall that way. Once inside, he could perform a safe shut down.

Oddly enough, this is not the first time I've heard of this access method being used in this subreddit --- at least one of those times involved an inebriated support person who was called in on their day off.

[-]

Blizerwin@reddit

Reminds me. Unannounced power test in Primary Server Room.

They cut the connection to the ups and did their check. Ups supposedly could keep power running for 30-40 mins. ... This number was 3 years old at that time and the power went down after 20 min. Worst, after seeing that everything went dark they replug and kill some IBM drives of an old host system we were still using back then.

Was an interesting day seeing half your VMs switching to red.

Oh and was my first on call duty after apprenticeship (Jan 2020 .. ironically I believe)

paishocajun@reddit

Did you ever find out how and why HR could get in unescorted?

krennvonsalzburg@reddit

HR always give themselves global access. They're incapable of not abusing their position.

RodanMurkharr@reddit

They are right, they do need to come in for termination.

Vartra@reddit

Not to the server room itself, no unqualified personnel should ever have free access to that, it's a quick and expensive way to cut a giant hole in a company's budget, ability to meet client needs, or just do the simplest task such as clocking someone in.

They can be escorted. They don't need unescorted direct access to technical areas.

RogueThneed@reddit

r/whoosh ?

Less_Author9432@reddit

But what if it’s the manager we’re terminating??

kwizzy2@reddit

Have known managers who should not have been given access…

HoochieKoochieMan@reddit

If HR is terminating the IT Manager, they hopefully have a Sr. IT Admin in the loop.

You have an awfully high opinion of HR’s willingness to share information….all too often the first time an assistant knows anything about their boss being fired is the company wide email congratulating them on taking over their boss’s role 🙄

The 3 times I've had to term my boss, HR gave me about 16-24 hours notice.
Partially to make sure I'd be there the next day to turn off all his accounts, and maybe also to see if I'm trustworthy enough to keep on.

Then you get one of the C-Suite to look more like they're earning their salary.

Kuddel_Daddeldu@reddit

The C-Suite should not have access to the data center either.

If the company is large enough to have a CIO or a CDO, I would expect them to have access. And if it's not, the CxO to whom IT reports can summon someone to open the doors for them.

Furdiburd10@reddit

Why would C suit access be a problem? Elon handled the datacenter migration of Xitter and nothing went wrong! /s

🤣

Yeah I wouldn't want C-suite either. A director if they're Peter principled and had a tech background previously, or a peer manager should suffice.

Moneia@reddit

Same. You'll turn up one morning to see that the datacentre has been replaced with a couple of Mac Pro's because they're beguiled by their Apple laptop

LordRael013@reddit

I'm curious about this one too.

LupusTheCanine@reddit

You have already posted the same story here.

paradroid27@reddit

He cross posted to r/ShittySysadmin, that might be where you read it

I read it a long time ago and I don't think I am on that subreddit. Besides https://www.reddit.com/r/talesfromtechsupport/s/96C8Ctbo25

StuBidasol@reddit

Yeah I recognized the story as well. The lack of Halon was the lock for me because I remembered thinking they were about to trip it.

Wells1632@reddit

Would have been better for all if they had tripped a halon system.

Ok, good find and better memory.

critchthegeek@reddit (OP)

It got deleted by the bot because I linked it to any same company story - just saw it was deleted and reposted

MultiFazed@reddit

Doesn't look deleted to me: https://www.reddit.com/r/talesfromtechsupport/comments/1jvfxlr/

mafiaknight@reddit

Looks like you had a double-post 159 days ago, and the duplicate was deleted

Inquisitor_ForHire@reddit

We had this happen. Our UPS was supposed to be wired so that if we lost power the UPS took over and the generator fired up. However something wasn't quite wired correctly. They put a smoke puffer on a the sensor, and immediately the entire data center went silent as a tomb. UPS was still happily sitting there hording all it's power yet providing none of it to the servers. We lost a few hard drives, but nothing unrecoverable. The dude doing the smoke test went so pale I thought he was dead.

And to add insult to injury, the backup generator fired up like it was supposed to and started adding more power to the UPS so it could keep it all.

Someone had some explaining to do!

Jonathan_the_Nerd@reddit

UPS was still happily sitting there hording all it's power yet providing none of it to the servers.

It was busy powering the most important piece of equipment in the room. /s

Well in its mind yes! :)

Techn0ght@reddit

HR investigated themselves and found they did nothing wrong. Probably blamed MIS for not properly training HR in areas they shouldn't be.

fresh-dork@reddit

training can be interpreted flexibly. demanding an audit and explaining the painful consequences of them ever being found in the server room is a starter

But now they'll drag you in for a warning about "bullying" because you hurt their delicate feelings

jeffrey_f@reddit

Halon was fun because you needed to leave or you would likely not survive the halon dump.........Earlier fire suppression was an immediate water bath with what would usually be a black watery substance from the stagnant water in the sprinkler system.........and an odor you won't forget for quite a while.

C-Suite meeting to explain the reason the company will be dead in the water for at least a few hours.........

Aperture_Kubi@reddit

Why is HR doing facilities work though? Sounds like an HR violation.

Woodfordian@reddit

It's even more fun when HR sets off the halon system by demanding a test of an alarm circuit. Normal channels would have stopped him.

The box was clearly labelled as not part of the general fire system. He was a self-important unemployed person later that day.

AngryCod@reddit

Good lord. My halon system costs $60k just for the tank refill, forget about the service call, cleanup, and equipment replacement.

annemg@reddit

Our IT manager did this, luckily it has just been installed and the system hadn’t been charged yet. He did it right after the installer had said “don’t touch this button, it will set it off.”

p.s. it was an amazingly cheap clean up at only $60k

Well that's the beauty of ozone depleting, people suffocating and toxic (at quantities used in data centers), clean fire suppressant - halon. It would have been way more expensive if the system used water 😅.

SourcePrevious3095@reddit

Edit explains it all. I was thinking katma farming repost bot for a second.

Fubared259@reddit

Our server room once had a sprinkler system in it. They realized that was a bad idea only after realizing the room next door was the main breaker room.

Terrornator@reddit

Why are you reposting the same story you posted 5 months ago

https://www.reddit.com/r/talesfromtechsupport/s/pg9HHINIjo

Turbojelly@reddit

I think this story has gone round the fire detector testers. Last time we had someone in testing, they asked us to make sure they could test server room and switch cabs without setting off any extra precautions.

handlebartender@reddit

Vaguely similar story. This would have been in the early 1990s.

I was in the modest computer room, having a chin-wag with the operator. The company was slowly transitioning from older systems to more modern ones. This room had a small HP/UX system, a handful of Novell NetWare servers, and I’m sure a few other meaningful things. And an absolutely huge UPS system that none of us knew how to operate, nobody told us we would ever need to think about, etc.

Chit-chat interrupted by a knock on the door. Building admin, needed to check the heat sensors. Had something like an incandescent bulb in an open-ended enclosure on the end of a pole. We said sure, he went to work.

He dipped out right when we noticed things had gotten quieter. The operator’s terminal wasn’t plugged into the UPS, and the systems he accessed with it weren’t in that room. But the other systems were all down.

“Weird. Well let’s turn them back on.” And we started to realise that none would power up. We had a moment to think, realising the lights were still on.

One of the important managers came knocking, asking what the issue was. We eventually realised the UPS was to blame. But why? No matter, let’s just turn it on. Or off/on. Or something. Er… how do we do that?

More managers get involved. Increasing numbers of unhappy users. Who has the UPS docs? Hey, rumour has it that it’s connected to the fire detection system. More guesses. Escalation to building management, as the supposition was that they could see the problem and slip it back on at a control panel. More shrugging.

We ended up on a call with thrnUPS vendor to ask how tf to get this to resume power. Cool, instructions acquired.

Back to the server room with screwdriver in hand. We needed to pop open an access panel and disengage the motor that flips the switch, then proceed to flip the switch.

Systems begin to whir. Monitors blink to life. Optimism for the first time in what was probably a couple hours had started to flow back into us. OS banners, boot sequences, and at long last, the final points that show success, whether login prompts or otherwise. More checks, but things looked pretty promising. All-clear sent out.

Lots of questions around that UPS after that. I don’t recall the details, unfortunately.

standish_@reddit

some movement caught my eye.

Keep absolutely still. Its vision is based on movement.

falcopilot@reddit

I think I worked there...