The Flood Part 2:

We all stood momentarily in awe and shock. Stan is gaping up directly next to the torrent of water blackened by years of sitting in pipes, just waiting for the day when it is called to meet the urgent need to save lives. The other three of us were stunned in disbelief what still feels like minutes but what empirical evidence shows was only a few seconds. And now the fire alarm sounds and emergency lights begin to flash. Someone then reasonably suggests we find the valve to turn off the water, so we all bolt out the back door to find what would surely be the obvious water main. Out the back down and down a short stoop of a few steps we fan out looking for a valve. We pop up a few covers but nothing looks big enough to handle the volume of the water blasting out of the clipped sprinkler.

(This part was later related to me). Our boss, go with Al, hears more commotion outside his office and waits for a few moments for us to settle down. Hearing that we don't and now distracted from high stakes spreadsheeting, mail replying, ERPing or whatever boss stuff he was doing, Al pops out of his office to give us an earful. But he finds not a one of us there and instead finds a reverse geyser of water spouting from the ceiling, hitting the floor and splashing the nearby equipment and file cabinets. So he goes into action moving laser printers, document trays and other items away from the splash zone, all the while wondering what the hell just happened.

After a short period of failing to find the cutoff, we all return inside to see what we can do to mitigate the spread of the damage. We divide up. Most of the guys begin to go to each cube and remove any items on the floor and relocate the PCs, surge protectors, subwoofers, and other items as far away from the spray zone and spreading pooling water as possible. In the nearby breakroom, there is a large trash can that is conveniently on heavy duty casters. I remove the liner and push it under the gushing water. The water is crystal clear at this point, the latent brackish content of years of sitting finally purged from the system. It is also cold. I wait a few moments, let the can mostly fill and then push it outside the front door and dump it on the walkway into the parking lot. After doing this once or twice, I am soaked through. I take off my sopping wet shirt as it is getting heaving and binding me, pulling on me as I muscle a very heavy garbage can of water down a carpet floor and over two thresholds to tip over the curb. Somewhere around 10 trips of this, the local fire department shows up.

Al greets the fire department, assures them there is in fact, not a fire, and urges them to turn off the water. Out the back door they all go and finally the water is off. They shut off the alarms and the only noise is the dripping of the water from the filing cabinets. We all take a breather while the fire department sorts out what happened. The story above is told, more or less. When we get to the part about trying to turn off the water, we all ask where the shut off is located. Going out the back stoop, there is a handrail on the right. On the left is a large red pipe, rising about 3 feet out the ground. Hanging from it is a detachable handle. This is the shutoff valve we all missed. The fireman says it is a good thing we all missed as there is a several hundred dollar fine if anyone but the fire department uses it. We look around at the water logged printers, cabinets, carpets and other items we can clearly see and all come to the same conclusion that the fine would be worth the price. But as I would learn, water gets everywhere and it likes to stay there.

Sometime in all this we begin to look around more of the disaster zone. The nearby walls, carpet and filing cabinets are splatter painted with black gunk from the initial blasts from the pipes. The carpets are soaked and the water is running slowly all around the floor. And it is running toward the data center. We badge into the data center and see that the Liebert is reporting that the humidity is climbing rapidly. We pop a few of the flooring tiles from the raised floor and there is a stream of water, slowly draining from the dividing wall toward the racks. It is a few feet away from the water sensing switch on the air conditioning unit. We know that if it trips, the aircon to the room goes off. If the aircon to the room goes off, all the racks will be powered down in the US East data center where many critical applications and services are solely hosted (real redundancy was harder and more expensive back then). So we scramble again to find something to prevent this. One guy is popping flooring tiles off to see exactly where the water intrusion is located. I run off trying to find anything to stop the water. In the restroom area I find the janitor closet and there is a large box full of paper restroom towels. I bring the entire box to the data center and we begin to stuff paper towels under the raised floor in an effort to dam the water from progressing any more. After using about 3/4 of the box, we overcome the water's advance and block its access to the water sensing device. The humidity stabilizes just a few percent below the cutoff threshold.

In the chaos and melee of all this, the VP (or AVP, or whatever) who oversees the site, arrives. He had been on travel and just returned home that weekend. We were scheduled to meet with him over a nice lunch or something on Monday. Being somewhat junior and from a smaller office, I had never met him in person, but I was in his chain of command. So I finally met the guy ultimately responsible for signing off on my recent promotion, guy one step below our CIO, missing his Saturday family time and here I am shirtless, shoeless, soaked in an office devastated by water from a bored dude with a golf club. Winning!, right? Other local managers and service leads start trickling in to check out the damage and ensure their services are running okay. The office manager shows up and she knows all the firemen (its a smallish town) and makes comments in jest to me about dress code violations.

Plans are now formed about cleaning up what we can. The office manager asks about how the basement is faring? We have a basement here? Sure enough, we go outside, down around the hill and in through the back to the 'basement' which has been converted to engineering and design offices. Water has rolled down the walls from upstairs. Documents, electronics, equipment, furnishing are all soaked. More damage. That evening a few guys drive out to Home Depot or wherever to rent some wet/dry vacs. I go to the hotel for dry clothes and return to help with cleanup. The crew returns with a handful of rental carpet cleaners and we get to work sucking what water we can out of the carpeting. As we are running the vacuums, tired, stunned and wondering of the consequences, one of my friends (call him Pete) says

Pete: "Hey Sarlacc, look on the bright side!"

Me: “Oh yeah, whats that?”

Pete: "We're getting great training for our next job!" I burst out laughing, the tension somewhat released for me.

Stan the golfer: "Thats not funny man, thats not funny at all"

Me: "Disagree bro, that is funny because it's true"

We jokingly then made plans about opening our own Stanley Steemer franchise and similar type work. Stan did not appreciate our gallows humor.

In the end ServPro or a similar recovery company came in to remediate the damage. Dehumidifiers and fans were running in the place for weeks. Some equipment had to be replaced, one was a fairly expensive color laser printer. The data center never went down, but it did light the fire to add more redundancy to some key services and applications. I heard the total remediation bill was on the order of near $100,000, but I can't be sure. It surely was not cheap. No one was ever officially fired over this, but I'm sure it did not look well on future prospects for some individuals at the firm. By the time I left that company that data center was still in operation but more workloads were being moved to Azure and other cloud providers.

It has been a long time since this happened and some parts were left out and may be fuzzy. For instance I know we were driving in a crazy thunderstorm, maybe to pick up the industrial carpet cleaner from the remediation contractors. I'm pretty sure it was a Saturday when it happened as the office was empty, but I don’t think it was yet night time. Could have been a Sunday.

But there is one thing that remains abundantly clear to me and that is the Exchange 5.5 to 2003 move dialog was terrible!

TLDR: Golf club almost takes out data center. Exchange is all you will ever need.