Stupid problems require stupid solutions.
Posted by roflcopter-pilot@reddit | talesfromtechsupport | View on Reddit | 60 comments
Remember the heartbleed bug? That mean vulnerability in the OpenSSL library that made for quite some hectic days in 2014?
For our company, that bug came in a very unfortunate moment: The regulatory agency responsible for us had ordered a security audit just then - and passing it was critical.
In theory, getting all our devices in order for the audit's vulnerability check should've been a breeze. 90% of our user devices consisted of custom Linux thin clients, with a very streamlined deployment process: Get update files, push update to test group, validate it, deploy image files to production → all devices update themselves to the new OS automatically by the next reboot.
This worked great for all machines that were powered off, because when the users came in and switched them on, they updated themselves before login and were current for the audit the same morning.
Those that were left running by users at the end of their workday would've just required a remotely triggered reboot... Due to a freak coincidence, however, the current OS build suffered from a previously undiscovered bug that prohibited reliable execution of any remote shutdown command. So we frantically needed to find a solution for this, or we'd have a severe number of vulnerable devices left in the fleet!
Brainstorming within our team led to the conclusion that manually finding and rebooting those of the hundreds of thin clients that were left running was too time consuming and prone for human error. Some machines were also locked behind closed office doors IT had no key for. Then one of us had a brainwave:
"Hang on - aren't those machines set up with 'Restore on Power Loss = Last State' in the BIOS?"
You know what IT did have a key for? The main facilities room which housed the central power breakers for our HQ.
Powercycling the whole building did the trick: All previously running thin clients powered back up and fetched the update. By morning when the auditor came to us, 100% of our fleet was current with the heartbleed fix and we passed with flying colours.
Dustquake@reddit
I was so satisfied at the end because when you said "manually finding" and I thought "kill all power."
FrustratedRevsFan@reddit
shivers in manufacturing
LaundryMan2008@reddit
College did it too, students forgot to turn computers off and some even locked them for later so they popped the breakers for 5 seconds on both buildings and they were large buildings with 250+ computers in each one with sub buildings having 20+ in them which is what their IT regaled the tale to me
Lord_Lenz@reddit
This is the biggest "Did you try to turn it off and on again?" I've seen yet.
roflcopter-pilot@reddit (OP)
Throwing those big breaker switches was so satisfying, too!
Facilities was totally fine with it, btw - they just asked to safely disable the elevators before and for somebody to stand by on watch to confirm they actually stayed parked.
syntaxerror53@reddit
a breaker switch off/on soon stopped a mains-powered alarm clock that went off all morning on a weekend when was student living on site residences. next few mornings were peaceful.
Tattycakes@reddit
I’m picturing you like Ellie in Jurassic park, powering up the park 😂
wysoft@reddit
I always thought that "pump up the breakers" thing was a plot device for suspense until the first time I saw an air circuit breaker in use in a massive container loading crane.
The compressed air charge is there to basically blow out any electrical arcs that occur when the breaker separates, otherwise the arc can continue closing the circuit even after the breaker has opened.
lord_teaspoon@reddit
There was even a Unix system involved!
The_Real_Flatmeat@reddit
Good test for facilities too tbh. Not often they'd be allowed to turn off an entire building to check for issues
roflcopter-pilot@reddit (OP)
You're right! If I recall correctly, the HVAC system had acted strange after the last local blackout before. Thing is, our region basically never has power outages - probably a nice problem to have, unless you have to diagnose such an issue... Our powercycling of the whole building caused it to reappear, so they could get investigate it further then.
RayEd29@reddit
That's just proof of my mantra - "If it's stupid and it works, it's not stupid."
proxpi@reddit
43- If it's stupid and it works, it's still stupid and you're lucky
digitrev@reddit
Schlock Mercenary fan spotted
RayEd29@reddit
The 'stupid' stuff I've tried has worked entirely too many times for it to be luck. Nobody is that lucky.
Glint_Bladesong@reddit
Oh God I felt that...
NotYourNanny@reddit
I shudder at the thought of how many ways that could have gone sideways. The audit was probably more important than any of them, though.
roflcopter-pilot@reddit (OP)
It was. Not being compliant could’ve meant losing operational permits for the whole company, effectively grinding business to a halt until things were sorted out.
NotYourNanny@reddit
And that would be harder - and slower - to fix, too.
ManWhoIsDrunk@reddit
A couple of rogue UPSs could have caused some issues...
NotYourNanny@reddit
Depends on how long you leave the power off for, I guess.
roflcopter-pilot@reddit (OP)
Power was off for no more than maybe 5 seconds, since all we needed was a brief interruption. No worse than typical momentary outages during thunderstorms.
fresh-dork@reddit
KA CHUNK!
i'm assuming it wasn't the really big breakers where you have to wear a suit and have a buddy ready to hook you away?
roflcopter-pilot@reddit (OP)
Correct, to toggle the main supply breakers running into a building lot you need the electrical supply company here. They aren’t even accessible yourself.
What we toggled were the (still kinda big) main circuit breakers of which there were three per floor and left/centre/right subdivision of the building iirc.
JereTR@reddit
Reading this, before getting to the last couple paragraphs, my thought was "why not just power cycle the entire building?"
I'm happy my intuition meshes with your thought process to fix this.
nymalous@reddit
Same.
I__Know__Stuff@reddit
Me too.
Equivalent-Salary357@reddit
Elevators! Someone was thinking that day/night.
Stryker_One@reddit
And luckily, no arc flash.
Turbojelly@reddit
Click clack, went the breaker switch, taking a load off your back.
CanonFodder_@reddit
More like BANG when the breaker is opened and a CLUNK when it's closed again haha.
But yeah I like the term taking a load off for them haha.
Available-Topic5858@reddit
I needed to do this once to a piece of equipment on board a nuclear submarine.
For stupid reasons our little company that normally made bubble detectors used for medical used (could detect bubbles within a tube from the outside) was told by the Navy we had to build a level detector for the SeaWorld subs. They used the same one on the Virginia class.
Yep, our box would make sure there was enough water for the nuclear reactor, because as we all know "you can't put too much water into a nuclear reactor. "
So there i am, civilian contractor in the bowels of the Virginia. Our unit there... not following its settings. Motor not turning on when they water hit a certain level, despite what the display was reading. I assumed that number was being stored two ways, as an integer, and something else to display. A reboot would re synch them.
Took a while to get permission but the reboot worked.
songbolt@reddit
Die Hard scene: "Shut it down; shut it all down now!"
eatingthosebeans@reddit
I was thinking of the American Dad ”Family Land" episode.
Mister_Bishop@reddit
Cue "Ode to Joy" as the computers all reboot properly and update.
parrukeisari@reddit
Sometimes in life you come to a point where regardless if your problem looks like a nail or not, all you really need is a bigger hammer.
spiritsarise@reddit
And if your company were distributed in many buildings scattered around a small city, you would need the biggest hammer: Blackout Springfield!
Notmydirtyalt@reddit
Turns out those substations attacks weren't grey hats or a test run for a terrorist attack, it was just Steve from IT who needed to reboot 3 remote sites in town he didn't have the keys to.
eatingthosebeans@reddit
Fun fact,
A lot of small transformer stations or landline distribution boxes, use the exact same keys, as commercially available server-racks.
Ich_mag_Kartoffeln@reddit
"As the size of an explosion increases, the number of social situations it is incapable of solving approaches zero."
Gambatte@reddit
...and that would be wrong.
db48x@reddit
FAMILICIDE!
wrincewind@reddit
but expedient!
ahazred8vt@reddit
Maxim 6: "If violence wasn't your last resort, you failed to resort to enough of it." -- The Seventy Maxims of Maximally Effective Mercenaries
Xillyfos@reddit
This is so satisfying to read. I love brillant ideas like this that suddenly just solves the entire problem. The feeling you get when you suddenly see the solution in your head is priceless.
lord_teaspoon@reddit
I am one of many independent inventors of the process of getting every machine in the building to pull a new config from DHCP by power-cycling the switches. My boss didn't believe it would work and had already started the manual process, but told me I was free to try it. By the time he checked the third machine it was in the new subnet. Very satisfying.
ThunderDwn@reddit
"Hello, IT. Have you tried turning the building off and on again?"
DimensioT@reddit
I remember Heartbleed.
It affected a production (albeit noncritical) system that my supervisor had set up. He was aware of the issue but as it would require essentially rebuilding the customized setup he was "too busy" to fix it even as Enterprise Security was coming down on affected systems.
One day when he was out I took it upon myself to upgrade it on my own. Took half the day.
andynzor@reddit
sudo sh -c 'echo b > /proc/sysrq-trigger'is my go-to solution.firedraco@reddit
That's some thinking outside of the (computer) box!
sgt_oddball_17@reddit
As I always say, every problem has a Layer-1 solution.
ManWhoIsDrunk@reddit
If the corporate site is big enough, you can even call the power company.
RayEd29@reddit
I've had to reboot a computer, I've even rebooted a network. You, sir, have set a record with rebooting the entire building!
sgt_oddball_17@reddit
Legendary
alaorath@reddit
Reminds me of the old IRC chat joke:
SevaraB@reddit
Ha- as soon as I read “remote power off,” my brain went “ya know, the breaker panel is the ultimate remote power off, and the CISO can deal with any ‘VIPs’ who get offended that their machines were powered off without telling them.”
Next up: smart breakers on timers (this is a thing). Their power WILL be cut every night unless there’s a documented business critical exemption that can incidentally be handed to the auditors along with a timeline for when the next maintenance window is for that exemption.
roflcopter-pilot@reddit (OP)
Smart breakers are interesting, never heard of those - sounds like a good idea, honestly, also from a fire risk/prevention point of view.
We implemented a different solution soon after this incident: Automatic forced shutdown after the last Citrix connection has terminated. Users cannot leave their thin clients running after work anymore this way. Gave our CISO more peace of mind, too, because that fresh boot next business day guarantees total compliance of both the thin client's software configuration and integrity, since every boot wipes them back to our predefined defaults.
SevaraB@reddit
They’re fantastic- smart outlets give you granularity but make you deploy and manage exponentially more hardware.
Imagine you’ve got a retail chain that doesn’t do “events” like midnight releases. Set up smart panels, smart locks, armored car pickup, and you can cut 2+ hours of labor per day per store with the simplified closing procedure (just clean and reset the store, count the cash, and done). No crazy electric bills from forgetting to kill the lights, no forgetting to lock the door on the way out or people who forgot their key setting off the alarm when they go back in (guilty), no more scheduling people til 10 when the store closes at 9, no more employees carrying bank bags in the middle of the night. If you can’t tell, I started my corp IT career in retail…
KelemvorSparkyfox@reddit
This is probably the best "turn it off and back on again" story that has ever been and will ever be. (At least until we reach Stage II, anyway.)
harrywwc@reddit
huh - when all else fails, reboot the entire building :)