In which the customer hoists themselves by their own petard - and a reintro

Posted by dennisthetiger@reddit | talesfromtechsupport | View on Reddit | 66 comments

tl;dr: Wanna hoist yourself with your own petard? Trychmod -R 777 /var on for size!

So, it's been a while. About a decade ago, I was the technical "triage nurse" at $UberNetworks. Well...I'm still there. And I've been promoted - more or less "senior attending physician. And after nearly eight years in this role here, I'm quite astounded by the number of people who make me wonder how they even got into our field....

This is one such story.

We're setting the Wayback Machine to sometime earlier this year. Don't remember when. Not important. But it was an afternoon ticket, lands in my desk, and I take a look: customer having a peculiar error when he tries to ssh into the box, and he wants to know why.

My usual technique for support is to stare at a ticket, see if we have any diagnostics, get them as needed, and we did all this and then I went in.

First check: a particular file somewhere in /var/*/ssh/* was apparently set world everything. This is momentarily confusing to me, and being I'm a Linux nerd, I already know that permissions do not just change on their own, somebody has to do it...

...in the immortal words of the sage Lister, "Aw, smeg!"

OK, clearly somebody did this, this sounds like an indicator of compromise. /var/log...oh this needs words stronger than smeg. Like shit. Shit's a good word for this. Look at all those permissions errors. Audit logs...nothing here, OK, it's probably CLI and don't tell me he just....

Check diagnostics...hmm, ntp is broke, nobody's answering hails, better tell 'im.

Look at commands, and being that our product is built on a Linux box, we have a full Linux install on there. I wish we had emacs^(1), but that's another story and we have nano, I'll live. But this means we also have 'history'.

I look at the output of history. Lots there, let's do a simple text search for chmod...ohhhhhh, shit, no he didn't. Oh, my gods, he did....

There it was, like platform double suede - exactly what I was hoping he did not do, and my hopes dashed, 'cause there it was, like disco lemonade.

In the history, with a username that I could only identify as being the customer contact's username just by the spelling of it, I see what I was afraid of. chmod -R 777 /var.

I stared at my screen in disbelief for five minutes, so we're going to pause the tape here and fast forward.

See, I've been dealing with computers since I was a child whose dad bought the TI 99/4A as the family home computer. I've been working in this field since 2006 in some way or another, with the exception of two years of college. I've seen people who I can't help but wonder if they got their A+ as the secret toy surprise in a pack of Cracker Jack. And in all that time, I had never seen somebody make a mistake that is the same grade of mistake as some wannabe skr1pt k1ddi3 who was trying to impress other nerds with l33t sk1llz. Until that day. When this guy, for whatever reason, altered the file permissions for - quite literally - everything in a Linux install that could be found in /var.

The reason the file permissions were changed were because this guy did exactly that.

My response and conclusion was thusly passed via email. Not five minutes later, I get a response - a request to close, sent as I was informing his sales team.

And then I check his ticket history.

Come to find out, he opened another case for the exact same problem right after he requested closure of mine.

Double you. Tee. Eff. Is this guy even thinking? No, really, is this guy even thinking?

Oh, it's on like Donkey Kong, motherfucker, you do not get away with pulling this kinda mommy daddy game^(2) horseshit on my watch.

Ticket intercepted. Pulled in, advised closing as duplicate, do just that. At this point, the sales team has been contacted. Oh hey, they're still here. Teams time! Passed word as to the update since this point, he nods, and he's gonna call the guy after he and I talk on the phone a minute. At this point, I'm wondering to my sales guy as to what exactly would even possess somebody to do just that, like what makes someone think this is a good idea?

A couple days later, I checked back in with the salesperson. He still had his job at that time, but it took a lot of convincing ot get him to admit it and stop denying that we were on to him. As best as we can tell, he was apparently doing it to prove some kind of point about the security of the VM installation - by doing the exact things you do not do. But after the Crowdstrike incident and my hearing that nobody actually got canned from that debacle, I guess I'm not surprised that this guy still had a job at that point. But at this point, I can't help but wonder if he is considering prospects in the wonderful world of convenience stores, because that - in my book - is a potential career-limiting move.


^(1) Yes, I know, ed is the standard editor...

^(2) What's the mommy daddy game? Well...if you have kids, you've probably played this game with them, and perhaps to some level of amusement. If you don't, it's the game where a kid asks mom for something, and on refusal tries dad.