Let’s Fess up to Some of Our Biggest Mistakes! Be honest, we’ve all made them.

OK - this one was not me, but I had to get things back up again after! This was an investment bank who hired an energy trading team (i.e. trading the spot price of electricity), let's just say in an ultra-time-critical situation. Ned head energy trader dude needed a feed to dirty power in the DC, to see what the actual grid voltage was doing, not our nice clean post-UPS power. The only feed for that in the DC was for the janitor to run the vacuum cleaner or whatever else they used for floor polishing! Dude wanted a permanent connection, not just plugged in to the outlet, and he wanted it done NOW - mid morning on a fast-trading market day. So the duty electrician needed to throw off the janitor circuit, then sawzall through an armored pipe, put in the junction box, and then wire into dude's trading monitor. The problem was that there were two armored pipes coming up alongside each other in a service corridor, then through the wall into the DC. Not clear what the second went to, as only one came out the other side. Dude screaming at the C-Suite by now about how much money he was losing by not having this thing plugged in, so the electrician made a reasonable guess on which one ... and "poof" off went the entire DC. Turned out that the conduit he took out went to one of the EPO (Emergency Power Off) buttons, but somehow it did a right angle in the wall in a way that made it look like it was the janitor circuit. And it was about this time that we realized that we had always shut down the DC in a very specific order for DR tests, and then restored it in the exact opposite order, networks, disks, core switches, DNC, DHCP, AD, etc, then into the infra systems, then the GL / AR / AP systems, then the middle office, then the trading systems. Done and tested dozens of times. But on this day, with the DC down at 10.30am and millions being lost per hour from being out of a fast-moving market the C-Suite insisted on calling an audiable and prioritizing trading first. Turns out this was a perfect recipe for chaos, with systems coming up with dependencies missing everywhere, and it basically took us 2x as long to restart than had we done it the tried and tested way. As a penalty for this, we had to develop and test restart scenarios for outages at any time of day, where different systems need to come up depending on what's time critical at that point. Massive work, and at least a dozen full weekends to develop and test the new procedures ... all for a freaking raw power feed!

My best ever (25 years ago!) was a big upgrade on a production VAX cluster, where for some reason we could not do a rolling upgrade. Shut down, split all the OS disk mirrors, boot each system, do the upgrade, check out individually, shut down, rinse and repeat down the whole cluster. Then bring the nodes back up into the cluster one by one. 12 hours later - all good and checked out. Last job - shut down, reintroduce the mirrors, and then reboot and leave the mirrors to converge again. The fuckup - I re-mirrored from the original disks to the upgraded disks, so the systems all came back up again as if those 12 hours had not happened. DAMMIT!