One of my previous roles was as a DBA for an e-commerce company. One day I was plugging along turning coffee into code when all Hell broke loose. Our marketing team decided to launch a "personalized" email campaign without consulting IT first, or even consulting anyone, really.

Out of nowhere, suddenly our servers started screaming at a pitch I don't ever want to hear again in my life. CPU usage spiked to 100%, and queries slowed down to zero. My first though was that we were being hit by a DDOS attack. What I found was far more facepalm-worthy.

The marketing team had written a query to send personalized emails to our entire customer base – all almost 5 million of them. Their query pulled data from nearly every table in our database, joining them in the most inefficient way possible. The icing of the cake was that they had set it to run every 5 minutes. It was later described by my senior to the bosses as like watching someone try to empty the ocean with a teaspoon, only to refill it with a fire hose every few seconds.

After some frantic calls and a lot of explaining (with technical terms I'm sure they didn't bother even trying to undersatnd), we managed to get them to pause the campaign. It took three days of optimization, index creation, and query rewriting to get their personalization working without bringing our entire infrastructure to a standstill.

The silver lining? Management finally approved our long-standing and often-denied request for a separate analytics database. Sometimes, it takes a near-catastrophe to get the resources you need