That time we accidentally bricked the CEO’s parents’ clinic network
Posted by CheckTec00@reddit | talesfromtechsupport | View on Reddit | 22 comments
I’m doing an apprenticeship at a company that manages networks for medical practices. Both our office and all the practices we support run on Unifi gear. One of those clients just happens to be the CEO’s parents, whose clinic is literally right next door. Their network is set up behind our office network.
One day, a colleague was tasked with setting up a demo server rack. Plug a laptop into the Unifi Dream Machine via LAN, WiFi off just to be safe, load up a backup image, add it to Enterprise Management, done.
Except… not done.
After the backup was supposedly restored, we disconnected the LAN and tried to reach the UDM’s web interface through the management portal. But it just didn't appear. So we kept poking at it, scratching our heads over what was wrong.
That’s when the clinic next door, the CEO’s parents’ clinic, suddenly lost their entire network.
Turns out the UDMs web interface we’d been happily messing with wasn’t the demo unit in our rack, nor the one providing internet to the rack from our own office. Nope, we’d somehow managed to connect straight into the CEO’s parents’ live production system which was also conveniently named exactly like our backup, so we didn't notice, and pushed the backup image there.
Needless to say, nobody was particularly amused.
Since that day, we use a separate Unifi account which can only manage demo and other clients networks, not the company network or that clinics network.
Dom_Shady@reddit
Don't be to hard on yourself. This wasn't your fault - the namegiver dropped the ball here.
4rd_Prefect@reddit
There are 3 difficult tasks in IT: 1) Naming things 2) Counting things
itenginerd@reddit
LONG ago, back when we used to name things actual names, I worked at a place where they named their servers after people in the bible. So there I am, working on their systems, when I pull up their AD, look at the DC list, pause for a moment, and ask 'so..... now, who exactly promoted Judas to be a domain controller?'. Turns out Peter died of a bad power supply or something, so they promoted Judas. I still chuckle about it on occasion.
randybear00@reddit
Our Sun web servers were named after planets, so of course Uranus was always the butt of our jokes.
Floresian-Rimor@reddit
I miss remoting in to Hogfather…
DarkVex9@reddit
There are 2 famous challenges in programming:
Naming things
Off by one errors
NewUserWhoDisAgain@reddit
Backup
Backup1
Backup12
Backup 123
Backup 2
Backup 1204
Backup 4
Backup 5
Backup 6
Where's backup 3?
Well, you fool, its right there! "Backup123"!
Dom_Shady@reddit
:)
I would add a fourth: administrating things
atomicsnarl@reddit
Ah, the 25/75 rule. Design and coding take up 75% of the time available. Debugging takes the next 75%. Then comes documentation.
You get the idea.
JoshuaPearce@reddit
It's worse with medical software. The things you listed there become 25% of the total if you're lucky, the rest is doing checklists of tests.
skiing123@reddit
Stupid checklists, I get on the job and I was told they are all wrong. Then, why did we print a dozen copies in color to not use them. Stupid Epic rollout
JoshuaPearce@reddit
If the checklists for medical software dev are wrong, then you have to mark it off on a checklist and it's a whole bigger deal.
atomicsnarl@reddit
Therac-25 has entered the chat...
Dark54g@reddit
Man, that sucks. Sorry you had to deal with that.
Grillmeister5000@reddit
lessons were learned :)
OcotilloWells@reddit
Great lesson in risk management.
boildkitty@reddit
Haha, hard lesson learned.
NotYourReddit18@reddit
The first error was setting up the networks of two completely different companies to be directly connected, especially as one of these networks is a medical practice and as such most likely handling sensitive personal information.
The second error was the naming of the devices.
And the third error was indeed not having a separate account for demo purposes.
But none of those errors were your fault, given that they were made long before you started your apprenticeship and you're an apprentice, there to learn how things are done.
androshalforc1@reddit
Or not done.
bigthe@reddit
Reminds me when i was testing stuff on our test server and proceeded to stop it after i was done... Except i had the wrong VM selected and instead stopped client's production environment.
maelish@reddit
I guarantee that you are not the first people to do something similar to this.
neomatrixj2@reddit
Nice