In which the customer hoists themselves by their own petard - and a reintro

Posted by dennisthetiger@reddit | talesfromtechsupport | View on Reddit | 67 comments

tl;dr: Wanna hoist yourself with your own petard? Trychmod -R 777 /var on for size!

So, it's been a while. About a decade ago, I was the technical "triage nurse" at $UberNetworks. Well...I'm still there. And I've been promoted - more or less "senior attending physician. And after nearly eight years in this role here, I'm quite astounded by the number of people who make me wonder how they even got into our field....

This is one such story.

We're setting the Wayback Machine to sometime earlier this year. Don't remember when. Not important. But it was an afternoon ticket, lands in my desk, and I take a look: customer having a peculiar error when he tries to ssh into the box, and he wants to know why.

My usual technique for support is to stare at a ticket, see if we have any diagnostics, get them as needed, and we did all this and then I went in.

First check: a particular file somewhere in /var/*/ssh/* was apparently set world everything. This is momentarily confusing to me, and being I'm a Linux nerd, I already know that permissions do not just change on their own, somebody has to do it...

...in the immortal words of the sage Lister, "Aw, smeg!"

OK, clearly somebody did this, this sounds like an indicator of compromise. /var/log...oh this needs words stronger than smeg. Like shit. Shit's a good word for this. Look at all those permissions errors. Audit logs...nothing here, OK, it's probably CLI and don't tell me he just....

Check diagnostics...hmm, ntp is broke, nobody's answering hails, better tell 'im.

Look at commands, and being that our product is built on a Linux box, we have a full Linux install on there. I wish we had emacs^(1), but that's another story and we have nano, I'll live. But this means we also have 'history'.

I look at the output of history. Lots there, let's do a simple text search for chmod...ohhhhhh, shit, no he didn't. Oh, my gods, he did....

There it was, like platform double suede - exactly what I was hoping he did not do, and my hopes dashed, 'cause there it was, like disco lemonade.

In the history, with a username that I could only identify as being the customer contact's username just by the spelling of it, I see what I was afraid of. chmod -R 777 /var.

I stared at my screen in disbelief for five minutes, so we're going to pause the tape here and fast forward.

See, I've been dealing with computers since I was a child whose dad bought the TI 99/4A as the family home computer. I've been working in this field since 2006 in some way or another, with the exception of two years of college. I've seen people who I can't help but wonder if they got their A+ as the secret toy surprise in a pack of Cracker Jack. And in all that time, I had never seen somebody make a mistake that is the same grade of mistake as some wannabe skr1pt k1ddi3 who was trying to impress other nerds with l33t sk1llz. Until that day. When this guy, for whatever reason, altered the file permissions for - quite literally - everything in a Linux install that could be found in /var.

The reason the file permissions were changed were because this guy did exactly that.

My response and conclusion was thusly passed via email. Not five minutes later, I get a response - a request to close, sent as I was informing his sales team.

And then I check his ticket history.

Come to find out, he opened another case for the exact same problem right after he requested closure of mine.

Double you. Tee. Eff. Is this guy even thinking? No, really, is this guy even thinking?

Oh, it's on like Donkey Kong, motherfucker, you do not get away with pulling this kinda mommy daddy game^(2) horseshit on my watch.

Ticket intercepted. Pulled in, advised closing as duplicate, do just that. At this point, the sales team has been contacted. Oh hey, they're still here. Teams time! Passed word as to the update since this point, he nods, and he's gonna call the guy after he and I talk on the phone a minute. At this point, I'm wondering to my sales guy as to what exactly would even possess somebody to do just that, like what makes someone think this is a good idea?

A couple days later, I checked back in with the salesperson. He still had his job at that time, but it took a lot of convincing ot get him to admit it and stop denying that we were on to him. As best as we can tell, he was apparently doing it to prove some kind of point about the security of the VM installation - by doing the exact things you do not do. But after the Crowdstrike incident and my hearing that nobody actually got canned from that debacle, I guess I'm not surprised that this guy still had a job at that point. But at this point, I can't help but wonder if he is considering prospects in the wonderful world of convenience stores, because that - in my book - is a potential career-limiting move.

^(1) Yes, I know, ed is the standard editor...

^(2) What's the mommy daddy game? Well...if you have kids, you've probably played this game with them, and perhaps to some level of amusement. If you don't, it's the game where a kid asks mom for something, and on refusal tries dad.

[-]

bucketybuck@reddit

I hope your code is cleaner than your storytelling. Cut out all the fluff dude.

[-]

ManWhoIsDrunk@reddit

We come here for stories, not log dumps.

[-]

MoneyTreeFiddy@reddit

Two extremes here.

One, where someone joylessly relates something that happened at work with excruciating detail, and none of it was worth retelling. As predictable as the alphabet. ABCDEFGHI, then J!

Then, there are things written with a lot of style that would work better on youtube, or radio. Stuff that begs for an editor's red pen. "~~Don't remember when. Not important.~~". (Then don't say it. -Everyone's english teacher.)
People should feel free to put whatever pizazz they want on their own story, but the final cut should be leaner than the first.

And, if you need to say "this will be important later", you aren't doing a very good job of setting the stage.
The Godfather doesn't tell you Fredo is weak, "and that will be important later", it shows him as a sickly baby, and he is outshined by his younger brothers.

[-]

dennisthetiger@reddit (OP)

You don't have to like how I write. If you have a problem with it, that's just your opinion.

[-]

MoneyTreeFiddy@reddit

Yep, just my opinion.

The part I hated worst lead me into my "stop saying this will be important later" rant, which you didn't actually do. You also didn't say "could of" or "would of" or "should of", but man, I really hate those, too.

[-]

dennisthetiger@reddit (OP)

To wit, I do use the line once in a while. Mostly for comedic effect, something in the way of turning a Chekov's gun into a shaggy dog. Or as a non-sequitur like they did with the watermelon in Buckaroo Banzai.

[-]

dennisthetiger@reddit (OP)

As it is, I'm kinda testing things for what I can describe as "power in apathy". Seems like if somebody tells you you're supposed to do it a certain way that they like, if you tell them you don't care, they tend to stop. One fool even deleted their own comment in this very post. =)

[-]

OcotilloWells@reddit

Yes. It's tales, not after action reports from tech support. We wanna hear about the man named Jed, who found the black gold, Texas tea.

[-]

ManWhoIsDrunk@reddit

Exactly. If i want to read quick summaries i got plenty of old, closed tickets at work...

[-]

OcotilloWells@reddit

Amen. And more every day. :-)

[-]

MuadLib@reddit

I'm doing both here

[-]

Jonathan_the_Nerd@reddit

I liked it.

[-]

wiseapple@reddit

Mmm - one point, ed isn't the standard editor, vi is.

[-]

dennisthetiger@reddit (OP)

I really don't care.

[-]

wiseapple@reddit

Nice. How classy of you.

[-]

Sceptically@reddit

ed context.

[-]

wiseapple@reddit

It's not though. It's a line editor.

[-]

Sceptically@reddit

It's a joke from the 90s, referencing an old man page. Thirty years is recent history.

[-]

wiseapple@reddit

*nix has been around since 1971, which pre-dates my interaction with it, but at this point I'm a gray beard. Linux on the other hand has not been around that long (relatively). Linus introduced the first Linux in 1991. It didn't really get into production environments until the mid 90s (which is pretty fast, considering). vi was introduced in 1976 and has been included as part of the base unix/linux OS pretty well since.

OP indicated that he knew ed was the "standard editor" and he's "a Linux nerd", "it's probably CLI" in his diagnosis. No one that deals with *nix is thinking "it's probably CLI". Of course it's CLI. That's what the OS is primarily. The GUI is for desktops, which isn't what OP described at all.

[-]

Sceptically@reddit

Referencing the "ed is the standard editor" joke does support the claim of being a nerd.

And there's a lot of "servers" out there with full desktop environments installed because they were set up by morons. And desktop environments often include GUI tools which can be used to break things.

[-]

wiseapple@reddit

True enough. I guess I'm too used to datacenter systems.

[-]

ac8jo@reddit

it's the game where a kid asks mom for something, and on refusal tries dad

My kids were warned long ago that we don't play the mommy-daddy game and that the penalties are very harsh. Y'all should consider something similar with your customers. Since you can't put them in the corner or take away the video games, charge them triple to fix. At least.

[-]

nl_dhh@reddit

Based on his role in this story, I think he's very much capable of taking away their video games.

[-]

dennisthetiger@reddit (OP)

Oh, I did better than merely taking his games away with that one. =D

[-]

Loko8765@reddit

Someone speedwalked into my office one day and asked if there was an easy way to recover from chmod -R 777 /

We said uuuuh easy no, not really, backups? Regenerate the server? And why? “Combined brain fart and typo” was the answer. At least it was a dev server.

But they owned up to it immediately and went looking for help (in the right place, my office being the daytime working space for the 24/7 emergency Unix systems support). Trying to cover up your error is a much worse problem, never mind accusing others.

[-]

Merkuri22@reddit

One time I forgot the "where" query in a SQL update command and overwrote all records in a table.

Like a good doobie, I went immediately to IT, told them I f'd up, and asked if there was a backup we could restore or something.

It was a small company, and IT was under the leadership of some VP that had -1 amounts of IT knowledge. They told her what happened (not sure why she needed to know, but she was the type of exec that had to know EVERYTHING), and her reaction was, "And she TOLD you?!?"

Yes, this VP was shocked not that I made a mistake, but that I fessed up to it.

🤷‍♀️

[-]

KelemvorSparkyfox@reddit

One time when running a purge process, I didn't forget the WHERE analogue, but I did make a typo in the range of serial numbers to be purged. These were 10-digit numeric strings, and I think my eyes were crossing at that point. Anyway, immediately after submitting the job, the queue of interface records to be processed was empty. That's odd, I thought, there were a few thousand in there, still waiting. Where have they gone?

I checked the report, gulped, and went to see my boss. I explained what I'd done, the impact, and then asked for permission to raise a sev 1 ticket with the support company to see if they could work some AS/400 magic and resurrect the records.

I've found that being honest in these situations, and presenting a plan for remediation, goes a long way to pacifying bosses and managers. Not manglers, though. All you can do with one of them is update your CV and hope for the best.

[-]

nymalous@reddit

AS/400 mentioned.

Memories of father's old job coming to mind.

[-]

GooderApe@reddit

Oh man, that brings me back.

Being told right away means it was an expensive training experience. As long as it's not a regular thing, it probably means a better employee in the long run.

Hiding it until it causes even more problems upstream is so much worse.

[-]

KelemvorSparkyfox@reddit

In my case, it wasn't that expensive. The support company were able to resurrect the purged records from the server's journal.

If I'd set the option to re-sort the file, on the other hand, the journal would have been cleared. That would have meant that a lot of production receipts from the manufacturing system would never have appeared on the sales order processing system. That would have been Very Bad, and possibly a P45-generating event.

[-]

dennisthetiger@reddit (OP)

See,, that's the thing, I expect people are going to be honest in our field. Unfortunately, my customer.

[-]

Loko8765@reddit

Users lie.

It’s a fact of life, a law of the universe.

[-]

dennisthetiger@reddit (OP)

Truth. And I keep forgetting, from my perspective, these admins are users.

[-]

bullwinkle8088@reddit

You can recover the default directories and files permissions (i.e files created during install) using RPM, but that will in no way fix the permissions of files created after install. SO say /var/log would have correct permissions, /var/log/messages *may, but any rotated log files would still be set 777.

As a means to get a server mostly functioning for data recover it's viable. It will not correctly fix it.

[-]

Valheru78@reddit

I would start with: find /var -type f -exec chmod 644 {} \; Followed by: find /var -type d -exec chmod 755 {} \;

That will at least make the machine usable again.

[-]

bullwinkle8088@reddit

RPM stores the ownership, permissions and SElinux context of every file and directory it creates in the package db. You can use it to restore them exactly as they were, with the limitations noted above.

The machine is trashed either way in reality, but the advantage of the RPMdb method is it can also fix chown screwups.

[-]

Valheru78@reddit

But slackware, arch or gentoo might have some issues, not to mention Linux from scratch ;) My method is a quick fix after which you can rebuild using any method that works on your distro.

[-]

Xlxlredditor@reddit

Yes, that might be the case, be any enterprise using Linux From Scratch as a server distro is just masochistic

[-]

SabaraOne@reddit

Every Linux user should build LFS once. Any Linux user who does it twice is a masochist

[-]

Valheru78@reddit

Very true 😊

[-]

Mr_ToDo@reddit

Any idea if there is a way to back up permissions in 'nix?

I know the quick and dirty for windows thanks to some fun adventures but I've never really thought about the other side.

[-]

Loko8765@reddit

I was going to say there is an option to cpio(1), but I must have dreamed it.

You probably have getfacl and setfacl, they will do it.

[-]

frymaster@reddit

I've done that before - I ctrl+C'd relatively quickly. There's an incantation you give to rpm (or one of the related commands) where it checks the permissions of every package-installed file. That actually got me most of the way there

[-]

Tom2Die@reddit

A bit late here, but I must ask: why did this user have root on the server in question? That seems insane to me...

[-]

dennisthetiger@reddit (OP)

With what this guy did, they gave him the keys to the kingdom. To be honest, I don't remember specifics - he may have set up external auth and got the admin bit by that point. Either way, he was amidst a deployment.

[-]

MountainMark@reddit

My best, as a semi-Junior admin, was an attempt to set all the ownership for a directory of random user files.
cd /home/
chown -R *

...hmm. That didn't get hidden files...

chown -R .*

All files starting with "dot". Golden.

Except...

.* (dot splat) matches .. (dot dot).

I just did, "chown -R .." which, when you're in "/home/" is the same as "/home".

chown -R /home.

Yeah - I recursed every home directory and changed the ownership of every file for every user.

It was a bad day but I was forgiven.

(Note: if you want to do this to hidden files, then it's ".??*" as a pattern.

[-]

spacecadetdani@reddit

... what?

[-]

AlaskanDruid@reddit

Ditto... this post doesn't make sense.

[-]

CheezitsLight@reddit

You Linux guys don't have vchk? it was in a BSD Unix back in the 80s. V7, System III and SystemV.

Makes a snapshot human-readable text file of everything you point it, such a /. if you have rights you can do a vchk - x and it crc checks and fixes all Permissions and even replaced bad files.

[-]

Mr_ToDo@reddit

Tell me, is it in modern BSD?

I was looking for VCHK and the only place I could find it was an old ass manual from 1983. All the modern Man resources and most of Google came up pretty empty

The 1983 one was a pretty interesting read though. It's a modified version of make used to repair files. It can check a bunch of stuff and even try and replace the files. The way it's written it reminds me a bit of SFC with a bit more user intractability.

[-]

CheezitsLight@reddit

It would spawn bin/take which downloaded data from a a 11-70 in Berkeley at Unisoft that did 68000 ports for my company and Sun , among others. I had the source at one time on 5.25" floppies.

Pert or python would be no more than a page of code now.

[-]

nl_dhh@reddit

If I'm not mistaken, it recursively sets read, write and execute permissions to every user, group and others to all files and folders under /var/

I get why that is unwanted from a security point of view, but can someone explain why this - by itself - would cause errors? I'm fairly new to Linux so I'm just trying to understand what the consequences are.

[-]

styphon@reddit

SSH specifically won't work if you set permissions to 777 for security. To prevent unauthorised use of it. So when you try to SSH in it'll fail to auth because the permissions were set to 777.

[-]

Mr_ToDo@reddit

It's always interesting to see software refuse to run because it's running to high. If I remember right in windows Add-AppxPackage won't run as System.

[-]

nl_dhh@reddit

Appreciate the response, thank you.

[-]

gargravarr2112@reddit

As the other commenter notes, it's basically impossible to recover from because of the sheer number of files that have very specific permissions set and have some kind of influence over the OS. Your only hope for undoing it is to have another clean system available as a reference.

With Linux, it's pretty easy to mix up chmod and chown (I do all the time). The latter changes ownership. It's actually far, far easier to recover from a bad chown than from a bad chmod because the system will at least be bootable if you set everything to be owned by root; however, many security-conscious programs will refuse to run with the wrong permissions set, which can essentially prevent you doing something as vital as logging in to fix the system.

And if you ever want to troll someone, chmod -x $(which chmod) will make most Linux admins cry.

Before hurting you, of course.

[-]

deeseearr@reddit

There's even more. The numeric permission "0777" will explicitly set read, write and execute permissions (4+2+1) for each file or directory's owner, group and everybody else (the three 7s). There are also three other permissions called the setuid bit, setgid bit and the "sticky bit". These aren't commonly used for user files but they are frequently set on directories which are used to write logs (/var/log) or store temporary files (/var/tmp), as well as binaries in /usr/bin which need to run with elevated permissions. Taking those permissions away won't be as immediately catastrophic as unlocking the secure files for ssh, but it will break the system a little at a time in ways which aren't quite as obvious.

Even better, many of the directories and files under var will have been created during package installation or later on while the applications were running rather than being unpacked as part of the packages themselves so the usual fallback of "check all of the package manifests and reset the permissions to what's in there" won't work. Cleaning up a mess of this scale is going to require either restoring from a backup, reinstalling everything or being very very thorough and checking each individual file one by one.

With that said, I am reminded of just how much incredibly important Enterprise(tm) level code I have worked with where the install guide includes the phrase "But first, run the command chmod -R 777..." because nobody could be bothered to figure out how to handle permissions properly.

[-]

OldschoolSysadmin@reddit

Actual conversation I've had, abridged:

"I need sudo."
"You don't get sudo."
"I can't get my job done without sudo."
"Open a ticket and have your boss sign off on it."

later

"I deleted /lib/ld_linux.so"

[-]

HammerOfTheHeretics@reddit

The first time I, as a dev, wound up with the root password at work the conversation went almost exactly the opposite way.

Me: "I need you to sudo something for me."
Junior Sysadmin: "Why don't I just give you the root password?"
Me: "I don't think that's a good idea. I don't really need that level of access, I just need you to do this one thing for me."
Junior Sysadmin: "No, it's fine. "
Me: "..."

I'm pleased to say I never screwed anything up with my unrequested access.

[-]

Sceptically@reddit

"And now you can't get your job done with sudo."

[-]

Immediate-Season-293@reddit

In 1998, I had a buddy going out for his master's degree in some computer science field. We worked together at an early web hosting/dev company. HTML and PERL mostly...

Anyway, he had classmates in his 300 level classes who had no fucking clue what FTP is.

I just ... sighed quietly when he told me about that.

[-]