A Big Oops Trying to Help Someone

Posted by Ok_Pomelo_2685@reddit | talesfromtechsupport | View on Reddit | 27 comments

A couple of jobs ago, I built a SQL query for finance that would fix sales discrepancies at the end of each month. These discrepancies were mistakes and typos done by the closing manager in our retail stores.

Finance would tell me what they were expecting based on the deposits. I was methodical about the process. I summarized all the individual transactions and dumped everything into a temp table for review before writing it to the production table.

I was comparing the production table to my temp to make sure what was already correct, was still correct, and what wrong, is now fixed. I discovered an issue, which happened, so I wanted to make a change and run the query again. Anyway, for some ungodly reason, I decided to manually type "DELETE * FROM TABLE", when I had the DELETE statement already build into my query and all I had to do was rerun it. Anyway, the table name I typed was the production table. I already had the left mouse button pressed to execute the command. The signal from my brain to my right pointer finger to release the button was already sent. I knew I screwed up as my finger was releasing the mouse button. I just deleted the entire production table.

Instant meat-sweats. I panicked for about 10 minutes until I was able to gather my thoughts. In that time, someone from finance called me letting me know that something was now really wrong with the sales data, which I already knew. I'm dreading telling my boss that we have to pull a backup. I then realized I could rebuild the summary table by using a few other tables. It took me about four hours to write a new SQL query to rebuild the table and test the output. The finance employee validated and all was good.

No one in my department ever found out. As far as the finance employee knew, there was already an issue that I was working on.

[-]

Ok_Pomelo_2685@reddit (OP)

Grest idea and I've done that with MSSQL DBs! This was an Informix database and I wasn't sure where backups were kept. I was also afraid to tell my boss what happened.

[-]

chrismsp@reddit

Good thing you know the database architecture like the back of your hand.. oh wait.

[-]

fiah84@reddit

Informix

oh god no

is that you Frank?

[-]

Ok_Pomelo_2685@reddit (OP)

🤣 I worked very closely with that name at this job, but he was in a different department, not finance though.

[-]

lollipop_anus@reddit

Because deleting a prod DB without ever notifying someone is never going to come up in a meeting you will have with your leadership right?

[-]

Ok_Pomelo_2685@reddit (OP)

Thanks! These are always fun stories to tell a couple of years later lol

Knowing the DB architecture like the back of your hand always helps in these situations.

[-]

Erestyn@reddit

This is why the product team love me at work.

"Oh no, I've deleted the company_data table!"

"No drama, just copy the table from company_data2. Don't ask questions, just know that decisions were made many years ago that you're about to profit from."

BlitzAceSamy@reddit

Ah yes, the ohnosecond

https://en.wiktionary.org/wiki/ohnosecond

EternalValkorion@reddit

TIL

Terrible_Shirt6018@reddit

If you've already clicked the dreaded button you can hold it, drag it off and release it. In most applications that doesn't send the command.

Unable-Entrance3110@reddit

Turbulent-Lab-4980@reddit

Do you all use auto submit statement? Executed DDL statements are not final until submitted.

But maybe it is an Oracle feature.

Tasty-Mall8577@reddit

i have a master Excel attendance spreadsheet. I did a Sort without selecting all the cells. I now have no idea where the “yes” and “no” should be. Not as fatal, but as painful!

Aazimoxx@reddit

But that is a case where Ctrl+Z should save you, right? 🤔

It would…if I’d noticed…before I saved it.

Doh didn’t quite cover it.

number__ten@reddit

I accidentally wiped a production database for a live corporate website. Thankfully i generally dumped backups on my desktop and had one that was only a couple weeks old. I was able to get that pushed up to get the website up and running again and we were able to call our host and get a more recent backup. The database mostly held articles and contact form submissions so it wasn't a huge deal.

Stryker_One@reddit

Ctrl+Z won't help you now, huh?

GovernmentPuzzled819@reddit

Imagine the chaos if it did work. Multi GB or TB delete and restore trying to happen at the same time!

Sounds like a fun time for the server.

KittyKong42@reddit

I love read-only contingency databases. Saved my butt a few times being able to 'compare' (reinstate) info sets.

CyberClawX@reddit

I'd shit a brick.

I once was updating descriptions, in our multilanguage description table. That table includes every description, label, and translation in our website. Everything from inventory, to menus refer back to that table.

And wrote my update like this:

UPDATE [DescriptionTable]
SET DescMedium = 'Ima dummass'
WHERE IdDescription = 1

I then only selected the first 2 lines, and pressed F5...

After 5 minutes of trying on every color of the rainbow for my face, I got up, warned my boss in the tiniest of tinny voices, sat down, and realized... wasn't that one of the tables with system versioned history enabled?

And that's the story on how I never ever started a DELETE/UPDATE/INSERT without a BEGIN TRAN ever again.

I decided to manually type "DELETE * FROM TABLE"

Anticipation of dread started when reading this.