Petards that hoist people, part 2: don't dismount the scratch monkey

Posted by dennisthetiger@reddit | talesfromtechsupport | View on Reddit | 40 comments

(Reintro: Support engineer at a company based in Seattle who is known for a tornado)

A common wisdom is to never go into maintenance without "mount(ing) a scratch monkey". There's a story to why they call it a "scratch monkey" involving a swimming primate, but the point is this - if you're going into maintenance mode, make sure you've tagged in/tagged out, signed off, opened the maintenance window, inform your users that this is gonna be a little bumpy, and you do the thing within that temporary arrangement because if you don't, you're going to blow up the pager.

Here's one such story.

A call comes in, we say hi and all, and he needs a remote right away. The colleague o' mine who owns the case is out that day. Line's noisy, so I tell him we can't get that going without a diagnostic file.

...which he...can't...get.

At this point, I started asking for a read on the errors he's seeing. It took me four tries to get it in a way he could understand - though to be fair, English is a hell of a language. But he basically started reading a bunch of daemon restarts.

...ayup, we're going to Teams.

Issue at hand is simple: after upgrading the operating system from an RMA replacement, an attempt to load the configuration backup failed for reasons unknown to me. The result is multiple daemon restarts.

We go in. I can't take control, so I watch the daemon restarts. Can't run the diag dump on the CLI, it requires a daemon that's not starting to actually be able to run. Reboot...um, well, it did work fine for all of ten seconds and then they could not get a thing started. I think now's a good time to roll back.

Talking somebody through command line is sometimes painful.

We get the CLI going, I tell him to run the diagnostic once more...and it burps. OK, let's start from the top. Let's roll back to the previous version. Run the command to change volumes and...

...hey. Hey, wait a second. Where's the other volume?

Again, three times asked - you started on this earlier version, where'd it go? Same cagey answers. And then I ask the big one.

"Did you delete that volume?"

They hesitated, and responded. Yes. Yes, they did in fact delete that volume. Somebody grabbed onto that idiot ball hard and decided it was not needed. And this is where a snippet of "Poor, Unfortunate Souls" from Disney's Little Mermaid starts playing in my head. In a fit of ignorance, they manually dismounted their scratch monkey. They blocked their fire exit. There was only one way to respond, and it required the placement of my forehead into the palm of my hand.

"I really wish you hadn't done that."

See, there are two ways out of this jam. One is to go in, review logs, and see if you can spot the bogey. This can take some time. The other is to simply bust out some bootable media and reinstall. And with this level of palpable inexperience, the decision was simple: take off and nuke the site from orbit, as it's the only way to be sure.

And I suppose it was good news for them that they could arrange bootable media and a trip to a data center.

I heard they called back, but that was the end of it from my perspective. Even so, this appears, once again, to have been a combination of ingrained ignorance combined with some unfamiliarity of the English language that tends to come up with when English is your second language - and at least one of these guys could not communicate without simplification (thus the thrice-repeated parts above) - and given that they called apparently not knowing how to boot and install despite instructions being in front, I suspect their greatest weakness was reading my language - the sort of weakness that can have you thinking Bellyvoo^1 is wee ired^23. So in my frustration, these guys have some sympathy for me - because my two native languages^4 are insane.


^1 Bellevue ^2 Weird ^3 Phonics, man, phonics. Not 100% accurate beyond second grade reading. ^4 English and bad English