Petards that hoist people, part 2: don't dismount the scratch monkey
Posted by dennisthetiger@reddit | talesfromtechsupport | View on Reddit | 40 comments
(Reintro: Support engineer at a company based in Seattle who is known for a tornado)
A common wisdom is to never go into maintenance without "mount(ing) a scratch monkey". There's a story to why they call it a "scratch monkey" involving a swimming primate, but the point is this - if you're going into maintenance mode, make sure you've tagged in/tagged out, signed off, opened the maintenance window, inform your users that this is gonna be a little bumpy, and you do the thing within that temporary arrangement because if you don't, you're going to blow up the pager.
Here's one such story.
A call comes in, we say hi and all, and he needs a remote right away. The colleague o' mine who owns the case is out that day. Line's noisy, so I tell him we can't get that going without a diagnostic file.
...which he...can't...get.
At this point, I started asking for a read on the errors he's seeing. It took me four tries to get it in a way he could understand - though to be fair, English is a hell of a language. But he basically started reading a bunch of daemon restarts.
...ayup, we're going to Teams.
Issue at hand is simple: after upgrading the operating system from an RMA replacement, an attempt to load the configuration backup failed for reasons unknown to me. The result is multiple daemon restarts.
We go in. I can't take control, so I watch the daemon restarts. Can't run the diag dump on the CLI, it requires a daemon that's not starting to actually be able to run. Reboot...um, well, it did work fine for all of ten seconds and then they could not get a thing started. I think now's a good time to roll back.
Talking somebody through command line is sometimes painful.
We get the CLI going, I tell him to run the diagnostic once more...and it burps. OK, let's start from the top. Let's roll back to the previous version. Run the command to change volumes and...
...hey. Hey, wait a second. Where's the other volume?
Again, three times asked - you started on this earlier version, where'd it go? Same cagey answers. And then I ask the big one.
"Did you delete that volume?"
They hesitated, and responded. Yes. Yes, they did in fact delete that volume. Somebody grabbed onto that idiot ball hard and decided it was not needed. And this is where a snippet of "Poor, Unfortunate Souls" from Disney's Little Mermaid starts playing in my head. In a fit of ignorance, they manually dismounted their scratch monkey. They blocked their fire exit. There was only one way to respond, and it required the placement of my forehead into the palm of my hand.
"I really wish you hadn't done that."
See, there are two ways out of this jam. One is to go in, review logs, and see if you can spot the bogey. This can take some time. The other is to simply bust out some bootable media and reinstall. And with this level of palpable inexperience, the decision was simple: take off and nuke the site from orbit, as it's the only way to be sure.
And I suppose it was good news for them that they could arrange bootable media and a trip to a data center.
I heard they called back, but that was the end of it from my perspective. Even so, this appears, once again, to have been a combination of ingrained ignorance combined with some unfamiliarity of the English language that tends to come up with when English is your second language - and at least one of these guys could not communicate without simplification (thus the thrice-repeated parts above) - and given that they called apparently not knowing how to boot and install despite instructions being in front, I suspect their greatest weakness was reading my language - the sort of weakness that can have you thinking Bellyvoo^1 is wee ired^23. So in my frustration, these guys have some sympathy for me - because my two native languages^4 are insane.
^1 Bellevue ^2 Weird ^3 Phonics, man, phonics. Not 100% accurate beyond second grade reading. ^4 English and bad English
Bcwar@reddit
OG Aliens reference .... One of us!
Left_of_Center2011@reddit
OG Aliens AND Die Hard! Thats a 10/10 from me
dennisthetiger@reddit (OP)
I was more thinking The Fifth Element, but either film is acceptable here because Bruce Willis.
asvalken@reddit
Well if she's so good, why don't they put her in charge?
Bcwar@reddit
Sounds like something Weyland-Yutani management would say ....
alpha417@reddit
Wouldn't have hired you unless you also spoke vulgar English
WackoMcGoose@reddit
Bonus points if you can curse in more than one language, ideally something Eastern European. O cholery, co do pierdolony jest, kurwa???
luther_crackenthorpe@reddit
Used to car share with a Polish girl - kurwa appears to do a lot of work in that language!
WackoMcGoose@reddit
It's the ultimate multitool across most of Slavic Europe! Although, Poland is the only one that spells it with a W... ask any of the Balkans, and they'll insist it's "kurva" with a V (which is technically more accurate to its pronounciation, only Polska does the V = W thing).
The cyrillic slavs have their own preferred curses (usually блять), but you'll sometimes hear them drop a курва as well...
dennisthetiger@reddit (OP)
Polyglots are at an advantage. Unfortunately, America's educational system makes "not English" an optional course.
Stryker_One@reddit
Does Klingon count?
EruditeLegume@reddit
HIja!
pawwoll@reddit
POLSKA MENTIONED?????
alpha417@reddit
czyli bóbr, kurwa!
dennisthetiger@reddit (OP)
That's kinda gotten lumped under bad English. I assure you, I am known to cuss like Roger Waters.
dickcheney600@reddit
I have a partition of unimportant things in which I also put "unknown" programs before I start them. I named that partition "scratch monkey" after hearing that old tale / joke
Frobbotzim@reddit
Aw man, the best thing f5 ever did (from my maintenance-window-dwelling perspective) was keeping multiple bootable partitions conveniently available for all your emergency rollback needs, no muss, no fuss, no grub (or not usually--I recall messing about with that in vi on the Viprion 4800's a couple of times, but they were special messes when it came to that).
What madman wrote a MOP that called for deleting the... before the... Oh, apologies, I was imagining that the guy doing the work would have been working from one of those, nevermind.
dennisthetiger@reddit (OP)
Just wait until you come across a one-slot instance in a cloud environment and want to upgrade. That's this, but...probably not as bad. Probably.
OldschoolSysadmin@reddit
Footnotes… you from that scary devil monastery?
jeffbell@reddit
Ref Error, footnote 23 not found
dennisthetiger@reddit (OP)
Glitch in formatting it looks like....
jeffbell@reddit
Sorry for the cheap joke.
I was trying to recall the banter of the Usenet days.
dennisthetiger@reddit (OP)
Eh, I getcha. =)
ShalomRPh@reddit
I haven’t posted in ASR in probably 20 years, but I still recall many of the regulars there.
Stryker_One@reddit
ASR?
ShalomRPh@reddit
Usenet newsgroup whose name anagrammed to “Scary Devil Monestary” (misspelling deliberate). This was where sysadmins came to hang out. It was deliberately misconfigured such that you had to manually screw around with the headers to get your posts to show up, as a kind of filter to keep the average lusers out.
dennisthetiger@reddit (OP)
That's a regrettable negative. By th e time I would feel I could qualify to go into the monastery, usenet was just a wasteland and hard to come by for free....
thekayfox@reddit
I can answer for them: yes, at least as far as I can remember.
Lumpy_Ad7002@reddit
Ignorance is okay. Ignorance combined with the arrogance is deadly. When you believe you are much more knowledgable than you are, when you believe you are much more knowledgable than everybody else? That's when crap like this happens
dennisthetiger@reddit (OP)
I know the condition well, "knows enough to be dangerous".
dickcheney600@reddit
Did you have trouble with reactivating the old, unsupported OS?
dennisthetiger@reddit (OP)
I wouldn't so much say it was trouble rolling back as it was a complete inability to roll back due to a deletion.
CaptainZippi@reddit
Upvote for the scratch monkey reference from me…
Reinventing_Wheels@reddit
It's an older reference, Sir, but it checks out.
I first heard the Scratch Monkey story when I was working part time as a technician while in college, [REDACTED] decades ago.
It was an old story even then.
millijuna@reddit
Ah, but what about the story of magic? I occasionally randomly install completely disconnected toggle switches deep in our systems labeled “magic” and “more magic”
CPlus902@reddit
Ooh, I love that story. I actually keep it bookmarked to refer back to when things start to feel a bit too mundane.
millijuna@reddit
Semi relatedly, we occasionally build systems that have a volume control. I ensure that the volume scales are always labeled 0 to 11.
mercurygreen@reddit
The Jargon File Glossary lives on.
zippy72@reddit
There's an old phrase about some people only speaking two languages: obscene and profane. I think I'd have temporarily turned into one of those people on that call...
12stringPlayer@reddit
Scratch monkey? Hell yeah, take my upvote, old-timer.