We need help, Server Room Air Con died... Chairman cuts a hole in the wall for a fan!!!!!
Posted by Outrageous-Boss6324@reddit | talesfromtechsupport | View on Reddit | 212 comments
Our Air Con recently died in our server room, luckily it's basically a separate room in our office, thus we used our office air con with the server room door open hoping to get our Air Con replaced.... our chairman saw the quotes and decided to instead KNOCK A HOLE in the wall and put in a big old fan... not a particularly sealed unit.
Now at this point my boss and the CEO were on holiday. Myself and the other IT guy tried to explain this is a very bad idea and were essentially told to stay out the way and let them do it. Now we have a hole in our server room wall and a fan,
My boss flipped his lid obviously but our Chairman said it works. Currently it's now hotter in our server than outside and we still have to use our office air con to keep cool and the chairman still thinks his idea is excellent... both my boss and the CEO can not convince him to replace the air con....
Also to note we are a damn national company with a bunch of location but all IT is done from the head office and the equipment in the server room is worth roughly 100K to replace IF we take our time shopping around for the best quote... its a damn mess!!!!
Ha-Funny-Boy@reddit
Years ago I was the 3rd shift computer operations supervisor for a company that had about 40 large HDDs. At around 9pm the 2nd shift shut down the online system that used the drives and shut them down too. When my shift came in around 11pm it was getting cooler in the computer room, so I turned the thermostat up. About an hour before the 1st shift came in I put the thermostat back to the original setting. The next day our manager asked me if I changed the temperature. I said I did because it was getting too cold, below 60F degrees. He said not to change it in the future.
So I didn't, but what I did was tell my crew the first one in the door was to go and turn all the HDDs back on so their heat would keep the room comfortable. We did that the rest of the time until the company moved from Los Angeles to Chicago.
Think 8 hours per day of 40 large HDDs spinning and just creating heat, what that added to the electrical bill as opposed to the AC system. Remember AC is not just cooling, it is heating also.
tsvk@reddit
Is the hole in the wall big enough for someone to climb through?
i mean if so, then the physical access to the server room is compromised, because keeping the door locked won't protect the equipment from tampering and theft.
oaomcg@reddit
just keep a Rottweiler in there
Langager90@reddit
Gonna turn into a hotdog once the room temps start climbing.
KnoWanUKnow2@reddit
Also, what's the fire suppression system in the server room?
Something like Halon won't work very well with a hole in the wall, and if there's people on the other side of that hole. The fire marshal won't be too impressed.
francis2559@reddit
Only 100k in server hardware, would they really have a Halon system?
samtheredditman@reddit
If the sales guy talked to the chairman directly, yes.
FredFnord@reddit
Nah, it would be water sprinklers hooked up to the bathroom faucet next door.
nullpassword@reddit
catches fire, halon goes off, gets sucked out, fan acts as bellows, play forge of servers.
PolloMagnifico@reddit
Oh shit I hadn't even considered the fire suppression.
created4this@reddit
You'll have to dodge the fan blades as you crawl through. You can kill the building power but the backup generator will start the fans turning again after 30 seconds.
PyroDesu@reddit
Or just... knock the fan out of the wall.
kuulmonk@reddit
This is NOT a Heist movie.
MikeSchwab63@reddit
Tango & Cash?
LemurianLemurLad@reddit
Not YET.
dragonfornicator@reddit
Not with that attitude it isn't
John_Tacos@reddit
This probably violates several clauses in multiple contracts.
NeppyMan@reddit
You're so close to the BOFH solution to this problem... Just have the culprit "demonstrate" the insecurities. Personally.
sekh60@reddit
Just put some chicken wire over it. It works for their garden, doesn't it?
Chakkoty@reddit
I think this is worth reporting to the fie brigade...then man is a safety hazard. And the rotary-hole too, I guess.
Scrapple_Joe@reddit
Horrifying, next step you'll want to put big blocks of ice over the servers to keep them cool.
fractalife@reddit
Reminds me of when I was really young, playing this obscure MMO that my GPU couldn't handle. I had the bright idea to put ice aluminum foil and hold it onto the heatsink of the GPU. Worked a treat, much better frames. At least, until it didn't.
The funny thing is that I knew better at the time, but my curiosity got the better of me. At least it confirmed my suspicion that the GPU could handle it if it had better cooling!
lookyloo79@reddit
Wait, what happened? Did the aluminum catch on fire?
fractalife@reddit
The ice melted, then condensed on the outside of the aluminum foil. I figured it would happen but did it anyway and tried to keep the water from hitting components. I was obviously not successful lol.
lookyloo79@reddit
Oh shit I missed the ice part. Are we talking about ice cubes wrapped in tinfoil?
fractalife@reddit
Yep!
lookyloo79@reddit
Hilarious. Thanks for doing the work so we don't have to
fractalife@reddit
No problem haha! I leave water cooling to the manufacturers now đ
I saw a LTT video where they used a water cooler piped through a CPU cooler block, and used that to cool the CPU. I remember them grappling with the condensation but don't remember what they did or if it worked.
I'm trying to dig up the video, but it turns out he's done a lot of videos on watercooling. Who would have thought? Haha!
If I find it, I'll update!
spdcrzy@reddit
Linus and Alex regularly up the bar for the craziest kinds of watercolooling shenanigans lol.
Sether_00@reddit
Check out Life Of Boris, he made water cooling with Vodka and empty bottle đđ
fractalife@reddit
They really do! Which unfortunately has the side effect of making it difficult to find a particular video from them about it haha. Especially when your only standout keywords are "water cooler" lol
Trekris@reddit
Did you invent the first liquid cooling system for computers?
ITstaph@reddit
No, cover the server racks in a ghillie fabric then put the ice on it. As the ice melts the suit will absorb the water and with the fan going it will therefore make a rudimentary swamp cooler.
/s
derklempner@reddit
No, you see, you use dry ice and then there's no melt to collect!
Just don't stay in the server room for any time longer than you can hold your breath.
Sm314@reddit
In a pinch this doesn't sound ridiculous, big block of dry ice with a fan in front of it. I've heard worse ideas..
spdcrzy@reddit
Yeah, that's actually a pretty reasonable option lol. Just dump a BUNCH of big blocks of dry ice in the room. Computers don't breathe oxygen, though. So whoever goes in will have to be in a sealed respirator unit to avoid suffocating as the CO2 begins to sublimate.
Mega_Dragonzord@reddit
But, you can mitigate that. Just put a bunch of plants inside there. Just like in 8th grade science. Plants breathe co2. See it works. /s
Jalero916@reddit
Except, the room has a giant hole in it... Into the rest of the office...
Moquai82@reddit
Into the chairmans office?
Sm314@reddit
Oh yeah, not harmonious to human life, but as a stop gap to keep the temps low.
ITstaph@reddit
Server room is already like that because of Chili Kevin.
tgp1994@reddit
That... actually sounds somewhat reasonable.
gamer10101@reddit
Congrats! You now qualify to be the new chairman
Techn0ght@reddit
A certain well known museum had a small room with a few racks with a rack top AC unit. With the air vent and drip pan not connected so it got increasingly hot and wet in that rack.
When they called us out I began unracking some stuff to troubleshoot and water literally poured out of the gear.
ascii4ever@reddit
Hope the budget will cover all the replacement hardware and downtime.
Frari@reddit
CEO should cover ITs ass. They should Email Chairman (CCing Board), outline the issues and consequences and that when something goes wrong (not if, but when) Chairman will be responsible.
Sir-Spork@reddit
When I hear stories like this I often think of the EPA guy who shuts down the containment unit in the first ghostbusters movie
HeadacheCentral@reddit
"Yes, it's true. This man has no dick".
HeadacheCentral@reddit
Not your fucking problem.
When it all goes to shit, point to the documented advice that not replacing the air-conditioning was a bad idea (you do have it documented, right?), then point to the chairman and say "He did it".
emby5@reddit
A company I worked at acquired another company. We go to check their server room, and see that they're running open windows. As in, the cooling system for the server room is an open window.
MoneyTreeFiddy@reddit
They need to uninstall that right away and replace it with Lennox. (An A/C manufacturer)
thgreatn@reddit
Logic checks out
deeseearr@reddit
Or they could just make much bigger windows, with a bit of a curve at the top.
"Oh, by the way, my cooling system runs Arch."
mercurygreen@reddit
Power down random servers in the middle of the day. "Sorry Chairman - it overheated!" (You should get a temperature/humidity logger if you don't already have one.)
Ddad99@reddit
The best CYA, and the one you MUST do, is a cost/benefit and risk analysis.
Alternative #1:Â Do not replace AC, and continue with current setup into the future; - High rist of catastrophic failure and associated costs, including cost of corporate downtime.and replacement of all servers and associated equipment, plus
Alternative #2: - Replace failed AC unit - Repair wall - Zero cost of replacing servers, etc. and low risk of failure.
Prohibitorum@reddit
Exactly, I'd be thinking on writing a detailed explanation/report of what the dangers of his 'solution' were. List the effects of the outage on the company, the costs of replacing equipment, the effects in fire suppression others mentioned, the compromised physical security, the costs of having to bring everything online, etc
Then send to everyone involved with read receipt and bcc for cya.Â
What a shit show.
Ddad99@reddit
Thank you for pointing out some additional risks.  It is extremely important to quantify everything with dollar amounts. I used to write business cases for just these types of situations and used monte-carlo simulations to quantify risks. I'm retired now, but you can easily find MC models that plug into Excel. Start at costs and then work the risk situations into the analysis. Don't make recommendations, just perfor. The analysis. Â
All you should have to do to get your management's attention is point out the high cost of failure and how very likely failure is to happen (complete shutdown and replacement of servers and associated equipment along with the AC). . The cost of downtime alone could kill the company. Look at Delta and the effects of the Crowdstrike shutdown. I would use that as the example in your report. They are suing CS and Microsoft for hundreds of millions of dollars.
Writing a business case is key. Describe what happened, what the current situation is , and the two choices for fixing the problem. Gather cost information and use hard numbers wherever possible. Â
Print hardcopies of the report and distribute to the executive team. Make a summary PowerPoint presentation. Sell your solution. If nothing is done and the inevitable happens...well, not your fault.
Prohibitorum@reddit
Offtopic question, but it has me curious: why do you type the extra spaces after sentences? I've seen other people do it (albeit rarely), and I don't understand the reason behind it.
betsys@reddit
Two spaces after a period is what we dinosaurs were taught back when we learned to touch-type on typewriters.
Prohibitorum@reddit
Gotcha! Must be annoying to try and break that habit. Thanks for explaining;)
betsys@reddit
Iâve got the hang of it. Now Iâm working on learning not to put a period at the end of my texts
Ddad99@reddit
I started doing that. Saves hundreds of keystrokes
Ddad99@reddit
Exactly. Also using the letter 'l' for the number '1'. No need to do that now.
NYX_T_RYX@reddit
Back everything up now, prepare restore scriots, then shut the door to the server room and wait for the inevitable beeping as machines shut down to prevent damage (I imagine some will be damaged, hence make sure your drp tools are up to date - either way you're likely to need them).
Oh and make fucking certain your objections to this wonderful idea are in writing - when it inevitably falls down, send a copy to said chairman along with quotes for the quickest replacements, and a copy of the A/Cv quote (in a dream world)
For real - make damn sure your drp is up to date then I'd shut the door and wait for it to fail tbh.
avrus@reddit
Hopefully everyone involved has a CYA folder and has emails from the Chairman disregarding your advice.
Epyon214@reddit
Hopefully the Chairman gets fired for incompetence and OP puts in a good word to get me hired in their place. OP should talk to HR and have the guys resume reviewed, probably faked the whole thing.
gryphonB@reddit
Of course they have the CYA folder, it's saved on the server! (/s)
mmoonneeyy_throwaway@reddit
This is when you BCC a protonmail thats dedicated exclusively to the issue!
RawketPropelled37@reddit
Why not just your personal Gmail?
rememorator@reddit
If something ever goes to litigation, it's better to not have your personal email involved, as you may be required to turn over access. Ditto with using a personal phone for work calls it email. If you have to turn over your phone they'll have access to every damn thing you have on it.
This is especially true if your work involves or is governed/regulated by a federal agency.
NotThatEasily@reddit
And thatâs is why I donât even check my work email on my personal phone. I made my boss give me a company phone if he wanted me to be able to check email, receive texts, or use teams away from the office.
Since then, my work phone has been handed into IT for them to make a copy of everything for FOIA and litigation purposes four times.
LongStrangeTrips@reddit
What I donât follow is if you have teams, outlook as work communication then why would you ever need to turn over your phone? That is all already available to the company whether they have your phone or not? Texts is another thing but at least in my case nobody would text each other anything work related as thatâs what teams is for.
rememorator@reddit
Late reply, but some examples of why is someone saving confidential/proprietary documents to their phone, engaging in communication with colleagues outside of official channels, or sharing company info with others via personal email or text.
Also, lawyers abide by 'trust but verify', so they can't just assume or take your word for it that you never did any of that and it's all just in Teams, etc. When something reaches the subpoena stage it's already serious.
NotThatEasily@reddit
My company uses text messages for a lot of communication and most people donât use teams for direct communication, because the nature of our job has us out where people donât have data, just a single bar (or sometimes none) which is just enough for texting.
My company often just copies phones of people involved or mentioned in issues and let IT and legal handle what does and doesnât need to be sent out.
Ampersandbox@reddit
Definitely the right decision to only do company stuff on the company phone, then!
topinanbour-rex@reddit
You assigned the s letter to your server, because it's start by s. Clever.
Strike_Thanatos@reddit
I mean, I'd be emailing the board members individually as well as major shareholders. They should be angry that the chair is overstepping his role and doing so against established best practices for IT, in a way that carries a risk of destroying company intellectual property and causing a fire in the office. This might honestly be grounds to remove said Chairman. Be careful to delineate the dollar value of the equipment in the server room as well.
jelly_cake@reddit
Not just the raw cost of the equipment, but the man-hours it would take to recover in the event of a catastrophe, plus the disruption it would cause to profit generating business units.
thetoffees@reddit
Don't forget the cost of blowing your SLAs since you said you're a national company. Oof - run fast and run far. This is going to blow up - or melt down.
jelly_cake@reddit
Wow, totally forgot SLAs - that'll bite.
SpongeJake@reddit
That's what happened at my workplace. As my career progressed there, I stopped caring what bonehead decisions they made. Just made it clear what needed to happen and the consequences of not doing so. Then sat back and enjoyed popcorn when the crap hit the fan
that_one_wierd_guy@reddit
and forward all the cya stuff along with photos of the chairmans handywork to the rest of the board
LillianIsaDo@reddit
This. Cause when it all goes wrong it will be anyone's fault but his.
thatburghfan@reddit
And even so, where it ends up is the Chairman says "Oops, my bad" and the IT team spends days on rotating shifts in a frenzy trying to restore service.
jeffbell@reddit
We had something like that happen back in the brownout days. PGE said we would be out for an hour. The generators kicked in, but they had only been sized for the servers and were not connected to the AC.
Someone decided that we could let the multi-day batch jobs keep running by propping the door open.
At about the 50th minute the disk heads started welding them selves to the platters and it took a week to restore everything from backup.
mike9874@reddit
A place I once was had a generator installed. Previously it was the good old UPS then auto shutdown.
The facilities team looked after swapping all the power to generator, we were told when out servers were done and so instructed by management to remove auto shutdown.
One weekend we had a call out for servers offline. Guy went in and the power was off, but servers still running. He couldn't get into the server room to see what was going on because his swipe card wasn't working.
Facilities guy used a crowbar to get him in, and destroyed the door.
Lessons learnt:
HPE servers power off before Dell ones when they get too hot. Cisco can power through anything
Check swipe cards for when they expire in the system, don't let them expire. Maybe even have a "break glass" card in the safe, just in case. (Yes, the system was fine, his card expired the day before)
The air con wasn't on the generator
The company made a person redundant who gets the notifications from the power company about power outages, and didn't update the contact
fgben@reddit
This is why I have generic operations@ or facilities@ type accounts for vendor contacts. Nothing should be tied to a specific person. Makes handover a hell of a lot easier when everything related to the account is in its own box and not buried in someone's general email box...
StudioDroid@reddit
I am a fan of tactical accounts. I don't understand using an employee's email for things like licensing and other subscription things. If they leave there is a lot of work to catch all the things sent there.
jonas_ost@reddit
Make sure card reader is on the generator also lol.
mike9874@reddit
They have a battery. That's what kept the doors locked on their magnetic locks
SeanBZA@reddit
You also get fail secure maglocks. Not as common, but are available, and will fail secure, even without power. Installed correctly they will be near impossible to release
jonas_ost@reddit
Not all does. We have 2 different door code systems on my apartment. 1 is only battery and if it dies you have to hold a 12 volt battery under it to temporarely start it to put in the code.
The other one goes on live current and has no battery in it.
But these are code locks and not cardreaders.
twopointsisatrend@reddit
I worked for a company that had a large generator. They had a service contract for someone to come out monthly for PM and testing. One day the power went out and the generator didn't start up. We looked at the generator and it looked like no one had touched it in years. I'm pretty sure the contractor had an uncomfortable discussion with the facilities and IT managers.
17HappyWombats@reddit
Places I've worked like that always had a monthly test. It's much better to discover problems on a Monday afternoon than whatever time Murphy dictates.
Hospital was the most brutal. Monday 2pm the head of engineering would turn something off in the power supply side of things, and if the hospital was affected he would put on his shiny, shiny boots of encouragement and get to work.
Current place we don't need to do that because we have a power monitoring system that reports 2-5 second outages happen roughly once a month without us needing to do anything. It's a feature!
Geodude532@reddit
I bet the redundant guy laughed his ass off when he heard what happened. Bonus points if he still got the emails and was waiting patiently for the outage.
ElvisArcher@reddit
Yep, been there. Exactly there. Buddy says "servers are still up!" ... CIO says nervously, "whats the temp in the server room?" ... everybody panics from there.
CEO drove a few towns over to find the biggest portable AC unit he could, and a gasoline generator. Everybody else got constructive with visqueen and box fans until the AC unit arrived. Then they just had to keep the generator fed with gasoline ... which was harder than it sounds since the whole city was suffering from power outages that lasted near a week in some areas.
Hurricane Katrina. Good times.
funkmon@reddit
Ha I remember stories; I was on vacation with the bad power outages in 03 or whatever and don't get hurricanes. I heard there were shortages and lines around the block
Tenzipper@reddit
Yeah, I had this issue when I was the whole IT department at a small company. "Why can't we just put a bigger duct into the room?"
FFS. No. This company would spend MILLIONS on a new sheet metal laser or such every year, but a few thousand for a server room A/C? I finally got it through their head by physically showing exactly how hot the air coming out of a rack is. At least I was able to get them to buy me a portable A/C while construction was happening, that I could use as a backup.
Fans are not a viable alternative, except in short term emergency situations.
p38fln@reddit
Bigger duct would work as long as the AC is sized appropriately. I worked for a library back around 2000, they put in a mini split for the server room but the buildings HVAC could easily keep up. They had an amazing (and probably insanely expensive) system which used water source heat pumps for each cooling zone and a massive cooling tower outside to cool the loop. The concern was the server room overheating the water loop during the shoulder months when it got below freezing outside at night but AC was still needed inside in the daytime. The server room heat pump just made free heat for the rest of the building once it really got cold since the excess heat from the server room was pumped into the same water loop the other rooms used for heat.
Tenzipper@reddit
No, it absolutely will NOT work, unless you have large commercial systems and engineers that can split things up.
Manglement wanted me to just put a big duct from the office A/C system into the server room. This was a small office building, and already had 3 systems, none of them particularly big. Think 3 large residential systems.
I pointed out that the server room system would need to operate ALL YEAR, in Nebraska, US. I asked them how they were going to like having the A/C on 24/7/365. Unless they were going to pay someone to set up a system to use the heat from the server room to heat the office, and vice versa. The costs are prohibitive, compared to just giving me what was NEEDED in the first place. Shockingly, as 2 of the 3 owners were finance guys, when I pointed out the costs, they quickly changed their tune.
Funny thing, those multi-million dollar lasers and other machines actually each had an individual cooling system for the PLCs and computers that ran them. I also pointed this out.
p38fln@reddit
Yep in the whole the system Iâm describing the server room still had its own heat pump. The âoutsideâ portion was in a mezzanine and the excess heat was dumped into a water loop for the rest of the building to use.
dank_69_420_memes@reddit
There's a reason HVAC techs and engineers can make so much, that shit is complicated.
Opheria13@reddit
Use your Chairman to plug the hole. Ensure that their head is facing into the hot side of the hole so that you can be sure they understand the many and varied reasons you donât want holes in the server room.
UneasyFencepost@reddit
Document the warning to them itâs stupid and go make sâmores on the melting server fire! They probably wonât taste good
Wise_Ad1811@reddit
Haha, let em learn the hard way. Close the door and watch the world (and your IT infrastructure) burn.
Trekris@reddit
Does this remind anyone of the post where the new manager messed with the thermostat of the server room and over the weekend the servers went into full meltdown but not before he sent an email admonishing the IT dept for the over use of the AC?
Trekris@reddit
This seems to be the dumbest idea. You compromised the Sercurity of the room with the hole and leaving the door open. You gamble with the integrity of the equipment.
Couldn't they have just purchased a portable AC unit at Lowe's or Home Depot until a more permanent solution could be found?
Scasne@reddit
He thinks it's a good idea because he "thinks it's cheap" so get it agreed that if things die before it's scheduled replacement it's coming from "his departments budget" if things slowdown for others accessing the network the same.
androshalforc1@reddit
$$$ Cost to fix the air con $ cost for weekly cleaning the server room as now excess dust is being sent in $$ cost in OT labor to get us back up and running after everything over heats $$$ cost to replace the parts on short term notice $$$$ cost for the business being completely down per day for likely 4 days minimum
After all that we still need to pay the original cost to fix the AC and fix the hole.
Flameon985@reddit
$$$$$$ remediate possible fire wall and retest room integrity
$$$$$$$$$$$$$$$$$$ if the place burns down and insurance doesn't cover because he broke the fire supression system.
fizzlefist@reddit
Now Iâm curious if it was classified as a fire wall. Cause if so, a hint dropped to the local fire department would get that shit sorted out fast.
Flameon985@reddit
With a gas system theyd be interested even if it wasn't. Google hfc 227ea
nullpassword@reddit
As a server room has a lot of electronics that are known to overheat if not cooled properly.. it probably should have been if it wasn't already.
geon@reddit
$$$$$ cost of getting all your data stolen because the physical security is compromised.
Newgeta@reddit
get it in writing and use the heat from the server room to prep some popcorn for the implosion to come!
MrJingleJangle@reddit
Every 10C rise in electronics temperature halves the life of the equipment.
HeavyMetalPootis@reddit
I'm tired and for a second I thought this was r/shittysysadmin.
eragonawesome2@reddit
Maybe I'm not understanding, your chairman is overriding THE CEO OF THE COMPANY? How? Is Chairman a role above CEO somehow? Is this some corporate thing I'm too private industry to know about?
scsm@reddit
This is how companies work. The CEO answers to the board and the board answers to shareholders.
Tarquin_McBeard@reddit
It is, but also it isn't.
The Chairman is above the CEO, in the sense that the Board appoints the CEO and manages the CEO's performance (which includes the performance of the company as a whole). But that's their role: overall big-picture strategy and performance management.
The Board is not supposed to be involved with day-to-day operational management. That's why the CEO is called the Chief executive officer. The CEO is at the top of the food chain when it comes to ordinary executive decisions.
The Chair doesn't have authority to just knock a hole in the wall and install a leaky fan. And the CEO doesn't need to convice the Chair that his plan was a bad one. The CEO has full authority to just rip it out and do it properly.
17HappyWombats@reddit
But the chairman's execrable decisions override mere executive ones...
RXrenesis8@reddit
A chairman of a company with $100k in IT assets... that's like half a rack max unless it's all second hand equipment...
All of this just seems like much ado about nothing.
The server room is probably an old janitors closet.
takesthebiscuit@reddit
The Chairman should have nothing to do with the running of the operations
Thatâs the job of the CEO - chief EXECUTIVE officer.
The chairman is to deal with long term strategy, investors and the likes
Anyway how is your CV looking
eragonawesome2@reddit
OOOOOHHHHHH chairman of the board of directors, that's the connection I wasn't making. I was thinking like, department chairman
Thank you!
Mdayofearth@reddit
In the Western corporate world, the Chairman refers to the Chairman of the Board, who is the head of a group that represents shareholder interests. When there is a separate person that is the CEO, the Chairman is never involved in day-to-day business, or how the business is run other than take part in approving budgets, expensive long-term business strategies, and f/hiring a CEO.
In Eastern corporate worlds, the Chairman may actually hold a lot of sway in the day-to-day operations of the company even if there is CEO. The Chairman position basically means they own the company, even if there are shareholders.
tempskawt@reddit
100K sounds like the off-the-shelf cost. When thinking of risk in this scenario, you have to lump in the configuration time and customer down-time as the total cost of the server room.
ElvisArcher@reddit
If the savings in reputation is worth more than the repair bill, then it was worth it.
pnlrogue1@reddit
Make sure your servers have thermal monitoring setup to shut them down before they get damaged. Shut the door and don't use the AC to cool them - just use the fan. When everything shuts down, make it clear that they've suffered a thermal overrun because the fans aren't enough. Keep letting it happen until someone gets the message.
intellectual_printer@reddit
Good plan till chair man decides to cut another hole in the wall for more fans.
Just keep lowering the thermal shutdown temperature..
i_write_bugz@reddit
This is the way. Makes a point without going full nuclear where everyone is screwed and then it really becomes your problem because now you're in charge of sourcing new servers.
jeremyledoux@reddit
Hopefully you guys have off-site back ups and a solid DR plan lol
TheDkone@reddit
this is a great time to make sure all your backups are in order.
eshuaye@reddit
And an off site copy
SuDragon2k3@reddit
We're paying how much for off sites storage?
Cancel it and get some portable hard drives or something....
SenseiTheDefender@reddit
Definitely test every restore-from-backup process that you can.
Skusci@reddit
As always, tell your insurance company.
Your mans just doesn't believe there is risk. Your insurance company does.
IRMacGuyver@reddit
Just start burning out CPUs and explain they're burning out cause it's too hot.
jonas_ost@reddit
As some1 who manages hvac for serverhalls.
Lol.
Even opening a door to the outside in winter times would not do much. Our cooling machines pull like 500 amps and we have 5 for one hall plus the smaller machines in each room.
You also need to think about humidity and dust...
hornetmadness79@reddit
Time to start migrating to the cloud!
20isFuBAR@reddit
Just let it go down and take out the company, then youâll be able to fix it quick.
momentum43@reddit
for a national company with multiple locations, are you not subject to access control reporting? do you not need to keep audit logs? whatâs to stop Joe Blow from Production who thinks he âknows a thing or twoâ about computers walking in and absolutely ruining your day?
harsh wording, serious questions though!
PandemicVirus@reddit
Smells like a pizza party in your not too distant future when all this dies.
andyspam1@reddit
Just keep the door closed. Even better if you have a written policy about access control to the server room to quote from.
"Policy says to keep the door closed, and you fixed the cooling issue. I'm not sure what you want me to do here?"
nico282@reddit
Oh well, it could work as a temporary solution
... not a temporary solution? Good lord...
daniell61@reddit
Op..... Are you from New Jersey?
JoeDonFan@reddit
Holy mackerel.
zggystardust71@reddit
Which way is the fan blowing? Did they think they could blow cold air into the room?
tryintobgood@reddit
I can't wait for the $100K replacement bill when everything ends up fried
Glass-Juice@reddit
As someone who works as an infrastructure technician in a very large data center this is appalling. I feel fortunate for redundancy.
gadget850@reddit
Sounds like my old company. Boss bought a building and remodeled it. We went to set up the server room and noticed the lack of air conditioning but there was a huge fan built into the wall. Boss stated that would be good enough. It wasn't. He had to buy a big AC unit and could not make payroll.
RoRoo1977@reddit
Switch off the servers âdue to overheatingâ and wait to see when he buys a new airco
GhostDan@reddit
they are gonna do that on their own ;)
GhostDan@reddit
Soooo.. there's drywall all over the place.. in a server room?
Yeah. That's not going to end well
DalekKahn117@reddit
Start overworking the servers. Letâs find out whatâs more expensive, an AC or a couple days of shutdown
althoradeem@reddit
this is where you start looking for other jobs. if the leadership refuses to accept your opinion as an expert it's time to just start writing letters .
Unsey@reddit
Boy it sure would be terrible if some servers ~were switched off at the wall~ switched off due to overheating issues....
MonkeyChoker80@reddit
Oh no no no noâŚ
This is bringing flashbacks of the time our facilities guy sold off the entire AC system to our server room three weeks before we were scheduled to move the servers outâŚ
Black_Death_12@reddit
Been there about 20 years ago. Was a "temporary" solution that was there at least a year. One weekend the $15 fan gave up the ghost and stopped spinning, but still had power. Caused some smoke, fire suppression went off, and thousands of dollars of equipment was destroyed. The river went down the hall, down the staircase and into the IT room downstairs, into the raised floor tiles. More damage and downtime.
Don't be cheap in IT, it will cost MUCH, MUCH more down the road.
oakwhiz@reddit
It's a great idea to pre-plan and pre-install those fans before these accidents happen. With security bars in the wall penetration of course.
Arch27@reddit
Well, that's going to cost them a lot more in the long run, what with having to replace all the equipment AND buy a new AC system.
Zavier13@reddit
Can we get an update as the Servers physically melt down, pretty please?
totmacher12000@reddit
Portable air conditioner? Or get an ac tech out there now. these are diy friendly
Leading-Force-2740@reddit
replace the chairman.
replace the air conditioner.
profit.
Reygle@reddit
Get the @#^% out of that job.
The_Razza7@reddit
Please update us when the inevitable shit hits that fan he put on the wall.
EyeBreakThings@reddit
So I assume these means access to the server room is compromised? Physical access to the servers is the first line of defense.
noeljb@reddit
Fan in wall should be blowing hot air out and sucking cool air from office.
farewell_traveler@reddit
Server Room is considered "occupied" 24/7, so now the office is "occupied" 24/7, as far as the HVAC system is concerned. The fan and compressor energy from the units running will increase energy consumption, and if there is a ratcheting clause in your area, the amount paid per kw/h will increase due to the increased demand.
100k seems a bit high to me, but I also know nothing about your server room, building, or area in which the building is located...
kagato87@reddit
You don't even need servers to hit that kind of pricing.
Core networking switches alone can get up there in price, and that's before adding some local domain controllers and a branch cache. Oh and the pbx. And the cameras. And that's assuming a remote site, nothing like cad servers or a virtual cluster (san fabric is very expensive).
A single twck can easily go well past 100k.
cactuarknight@reddit
Seevers can be very expensive.
Some of the units i used to work onj clocked in at 200k each.
Proprietary hardware is expensive.
velofille@reddit
This needs to be in /r/shittysysadmin lol. Omg what an idiot
zero_z77@reddit
We had something similar happen once. We had a single server rack shoved into a closet that was being cooled by the neighboring office's AC. It was just our backup server, DC, a couple switches, and some NVRs, so nothing too powerful. Fortunately there were air vents between that closet and the break room on the other side. So when the AC in that office went out, we put up a fan in each vent. Worked pretty well actually.
The reason this likely isn't working for you is because air cooling only works if you have airflow. A single fan isn't going to cut it, you need sone way to get cool air into the room, and get hot air out of the room. You need more than one hole in the wall for that to work.
But even then, this is just a temporary solution at best, the correct solution is to get the AC serviced/replaced. And if worst comes to worst, you can buy some cheap window/portable units from the hardware store and mount them to the wall.
PurplOrange@reddit
Doesnât sound fun for those working in that heat either.
Dreadpirate3@reddit
Best advice here is document, document, document. Make sure you get on paper that you and boss told him this was a bad idea. Now document how warm it is in the server room, the ambient DBA sound level in the office, and the value of the hardware in the server room. Also document how long it would take to replace one (or more!) of the servers if/when they fail due to the heat issues.
JamesWjRose@reddit
"dear Boss, this to inform you that WHEN, not IF this goes bad, I will not be helping clean up Chairman's mess"
Mischif07@reddit
I'm wondering if you work where I used to work :)
EmersonLucero@reddit
Air flow matters. Sticking a fan to blow in with out enough of way for the air to flow out is an issue. When my office server room AC would fail. Knowing we have a plenum building I would pop ceiling tiles to give the heat a place to go. (Our server room was not firewall off on all four sides, just one so air flow mixed with the common area air return) Then put a fan at the door to blow in. Lucky enough it was only for 4-6 hours at a time when it happened.
rUnThEoN@reddit
This method of cooling can work. Its basically a ventilated server rack but in big. The truth is you would need to suck in fresh air and duct the hot air outside of the building. That thermal transfer is best done via small pipes and a AC inside the room and Outside the building.
Responsible-End7361@reddit
Blowing outside dirt and dust on the servers can cool them yes. But there are other reasons to have a clean and isolated server room.
PolloMagnifico@reddit
Haha. The guy is literally describing air conditioning.
rUnThEoN@reddit
Let me phrase it differently: using filtered fresh air is a method used in some Data Centers.
oaomcg@reddit
that's not what was done here...
Responsible-End7361@reddit
Oh certainly.
But not what the chairman did.
rUnThEoN@reddit
It would work like a charm done technically correct. His theorie is good. His practical application 'blows'
Profvarg@reddit
Chairman: advice understood, hole now on the exterior wall as well
Yomat@reddit
Time to go over his head with an anonymous email. Heâs risking data security and productivity to keep HIS budget intact and away from prying eyes.
Equinsu-0cha@reddit
If you got the email chain saying bad things will happen and him telling you to shove it, Id say its no longer your problem. Bust out your fiddle.
filton02@reddit
Next he'll be turning the fan off at 5pm on Fridays, to save electricity.
wolfboy1988m@reddit
Why does this sound like the chairman trying to increase his bottom line by "cutting costs" at the risk of server integrity and uptimes?
MeriRebecca@reddit
last place I worked ended up buying 3 portable A/Cs for the 3 small server rooms. Vented them above the faux ceiling. And then put in a greenhouse fan to vent that through the wall up there into the rest of the buildings faux ceiling area...
It stayed that way for a couple years, until they finally sold the building.
withaph64@reddit
Iâm assuming the fan is pulling colder office air into the server room. Can you turn the fan around and duct the hot air into the chairmanâs office?
mcrib@reddit
Man I wish you'd name the company. They deserve it.
Horrigan49@reddit
That would be health and safety violation where I work As servers Under load are to damn loud to sit/Stand around for prolonged time period.
So employer would be mandated to mend that sht or face penalty from "osha" for noise safety limits variation.
Is There an option for you to play this Card?
kirby_422@reddit
A hole in an interior, or exterior wall? And blowing in, or out? I feel we need a few more facts to get the full depth of your (servers) suffering.
Warsum@reddit
Hehe itâs funny. I wonât say who I work for but we have major MPLS MP BGP network and site ânodesâ everywhere. One of our routers can cost 300k? ASR 9k with cards and 100 gig QSFPs.
Wanna know what we do when our five tons break? Send a tech to the location to open the doorâŚ..
AndiArbyte@reddit
you need to let the fan suck the air OUT the building.
This can help, should help.
Else, turn off servers and stuff not critical for use, and just piss of a few, not all.
There should be one outcharging the chairman.. A money giver maybe?
Time to Whistleblow?
Flameon985@reddit
Whoever handles fire safety in their jurisdiction might be interested, particularly if you they using a gas system for the server room.
Atlusfox@reddit
Just let it happen OP sometimes these types of managers need to feel the paim of thier decisions. Close the door. Double up on network traffic. Then when the blue smoke finaly leaves the sever switches it will be all on the bone heads.
joppedi_72@reddit
I really do hope you don't have inergen firesuppression in the serverroom.
bigoldgeek@reddit
Check your backups.
StuBidasol@reddit
Perfect example of "if you think a professional is expensive, hire an amateur."
Hopefully we get an update on the fallout if this.
IronSeagull@reddit
This is a ânational companyâ with only $100k worth of equipment in the server room and the chairman of the board is getting into day to day operations?
TR6lover@reddit
Chairman should stick to his lane and listen to the people that his company has hired in their areas of expertise. He will once the entire IT system fries and is down for 60 days being rebuilt - with new and improved HVAC.
wubbalab@reddit
Honestly, let it escalate until the whole dumpster fire ignites. You have tried to inform and fix the issue. But if the chairman thinks his solution is the best, he can take responsibility for the aftermath.
Word of advice: cover your ass, so this does not come back biting you. Also, if possible, try to have proper backups for data and the deployed environment so the systems can be reinstated as quickly as possible. Being the saviour in a situation like that can be beneficial for you.
Being prepared for disaster is key. Especially if it is foreseeable.
scyllafren@reddit
Shut it down and refuse to turn on, as it's outside operating temps. IT can override anyone for data and hardware safety. Put BIOS lock on it, so they can't turn on without you.
s-mores@reddit
Hope you have a CYA in effect saying your servers will just die and it will cost X to replace if you don't fix it... going weekly to ceo and board.
downtownpartytime@reddit
Make sure you have backups and a plan for when everything burns out
sarhoshamiral@reddit
I assume chairman isn't the head of the company here but just the local office?
Sounds like an email is warranted to CTO or equivalent. They may want to know 100k worth of equipment and data is at risk.
BrentNewland@reddit
Sound like Chairman of the Board of Directors. Reread the post, CEO can't change the guy's mind.
sarhoshamiral@reddit
That's how I read it first too but then it was said it was a branch office.
emax4@reddit
Start punching holes in his golden parachute.
HighOnGoofballs@reddit
Let it melt and laugh