Using router commands without knowing their purpose...

Posted by Inside-Finish-2128@reddit | talesfromtechsupport | View on Reddit | 52 comments

First, the situation: I was working for a Cisco reseller doing professional services in the early 2010s. Got sent to one of the local school districts to help them rearrange their Internet routers and deploy a "Firewall Services Module" in a Catalyst 6509. (IYKYK) My main contact from the school is stingy and won't share credentials with me, so I'm stuck walking around the datacenter and "sliding the keyboard over" many times.

We get the new "router triangle" running and to me, things are looking good, but alas I'm just doing the router CLI thing. Meanwhile, my contact has RDPed into his desktop (back at his office) and says to me "I have zero Internet at my desk". Hmmm...Internet routing is fine on the triangle, lemme go look further down the path and check the router at his office building. Check the default route (it's learned from OSPF) and it's coming from some random router somewhere else amongst their WAN. What?

I get logged into that router, and sure enough under the OSPF process there it is: "default-information originate always". What? Why? (For those unaware, that's basically telling that router to tell all of its OSPF friends that it has a default route, or a route out to the Internet, all the time. Hint: it doesn't.) I remove that command and my contact was jumping for joy, saying "WHAT DID YOU DO? THIS IS THE FASTEST WE'VE EVER SEEN IT!" Um, OK, I removed a misconfiguration from the router at site XYZ. He was in disbelief..."No, really, what did you do? How was that command bad?" So I explain it to him. As I'm finishing my explanation, a few of his coworkers who were on site that evening for some other work are tracking him down..."what did you do to the Internet, it's SMOKING fast."

That's when the "cover story" was born. He decided this needed to be kept on the down-low, so "we made a few adjustments to the dynamic routing and after about thirty minutes, those optimizations all came together".

Next morning, I head to the IT offices first thing for "day one support", just in case we missed anything the night before. No problems, just everyone jumping for joy with how fast things were working. I said "so, could that command be lingering anywhere else?" He helped me log into a few other sites and sure enough, it was EVERYWHERE. He finally gave me the creds, and I spent the next hour or two logging into every single router (one per school, probably 60-70 schools in that district and growing by 5 a year) to remove that command. Any time anyone came in to ask why the Internet was so fast, he jumped in front of me and rattled off the same line. He also did the right thing and told his team what we fixed, and said "be sure you take that out of any templates you have".

You guessed it...later that summer, the whole district went off the Internet because one of his crew brought a new school online and yep, forgot to take that command out of his template. Ah, the joys of using commands without knowing why they matter. (And I'll be honest, I'm surprised the whole network wasn't melting down every day or every week prior to fixing it.)

[-]

guitpick@reddit

We were too cheap for touch tone growing up, so I'm familiar with ATDP. I think I might have had a modem that would do pulse at double speed, but I may be conflating that with a phone I had that could do that (20pps instead of 10).

[-]

ThunderDwn@reddit

There was no such thing as tone dialing when I first started using a modem.

Hell, the first one I used was an acoustic coupler where you had to dial the number manually on the phone, then plug the handset into it when the other end answered.

300 baud. I could type faster than it could send! 🤣

[-]

guitpick@reddit

You're ahead of me by a bit. I got a 300 modem cartridge for my CoCo when everything was on clearance and used it more out of curiosity than need (this was around '95). I also picked up some weird Tandy modem at a yard sale that I didn't have the manual for (and it wasn't Hayes compatible) and never quite got it working 100%. I also paid 75 cents for a Kyocera acoustic coupler from a surplus electronics store, but never used it for anything. It looked like a nice one, though.

[-]

ThunderDwn@reddit

By '95 I was running a multi line BBS. I'd been playing with modems since something like '79 or '80 - my dad brought home a Silent-700 terminal from work, and I went through three rolls of very expensive thermal paper playing with BBS's in the first weekend before he banned me from using it and I went out and sorted the gear to plug a modem into my Vic-20.

Those were the days!!

[-]

diogenesNY@reddit

Around the same time, we had a silent 700 in my (grade) school connected to a small time sharing system. It was a DEC PDP-11 running RSTS/E.

I really used to dream of having a silent 700 terminal at home.

[-]

ThunderDwn@reddit

Around the same time, we had a silent 700 in my (grade) school connected to a small time sharing system. It was a DEC PDP-11 running RSTS/E.

That was pretty much what my dad used this for - remote access into a PDP-11/24 for work. I just kinda....appropriated it.

I remember going in to seeing this thing one weekend and being quite impressed. It might have been the cause for me moving into my current career.

[-]

Admin4CIG@reddit

I also had a Kyocera modem! Old, old times!

[-]

BeamMeUp53@reddit

I started on a 110 baud modem, with a converted IBM Selectric typewriter as a terminal. I have a 9600 baud modem somewhere behind the stuff that's behind the other stuff that's behind the useful stuff on the shelves.

Chocolate_Bourbon@reddit

I cannot count the times I've painted by numbers. Staring at coffee stained pages in a 3 ring binder, just following the recipe. I still do that today, but now it's a virtual document. And I now routinely create these instructions for others too. Funny how time is.

At least we've gotten past the point where people will call you and ask about any key. "It says now I need to press any key. I've looked on the keyboard and there isn't a key called 'any.' What should I do?"

Wise_Use1012@reddit

Which key is the any key?

Jonathan_the_Nerd@reddit

I read a funny story about that a long time ago. Apparently the space bar used to be called the "any" key. But then some company trademarked the phrase "any key" and successfully sued some other company for trademark infringement, and the upshot was that all computer manufacturers had to stop using the "any" label on their keyboards. That's why the space bar is the only key without a label on it.

Try telling that to your users next time they ask. See how many of them believe it.

jonrock@reddit

I appreciate that this yarn is slightly more believable because it correctly says that it would be a trademark problem, not a copyright problem.

himitsumono@reddit

No, sorry. The which key is NOT the any key.

TheKarenator@reddit

It’s a button that has a circle with a little line through most of it

Langager90@reddit

Depends on the date.

1st of the month is always "A", 4th will be "D", 12th is "L", etc.

"But what about the 27th on?!?" I hear your cry - just use numbers for these. "1" for the 27th, "3" for the 29th, up to "5" if you get that far.

It's a very intuitive, down-to-earth system. For exactly 3 people on the planet. The rest of us just press any button on the keyboard.

infostud@reddit

For use earlier in this month there are stickers you can put on keys.

Old_Sir_9895@reddit

Staring at coffee stained pages in a 3 ring binder

That's the problem with digital - you can't put coffee stains on it. Back in the day, theatre lighting designers had a saying: no design is complete until it has the brown ring of approval. With CAD software, one of the first custom templates was a coffee stain :D

Actually, I just realized that coffee stains are an indicator of reliability. If documentation looked new, if the binder was in pristine shape, that normally meant it wasn’t touched very often. Which meant it either was obsolete or incorrect or just worthless. Coffee stains meant it was used so often that folks didn’t care about appearances.

DrHugh@reddit

Next up...

ATZ
ATDT...

Dom_Shady@reddit

For those OOTL, what do these router commands do?

They're not router commands - they're commands from the Hayes dialup modem command set. "AT" is "Attention"

ATZ - reset to saved configuration

ATDT - dial the following number using DTMF (tone) dialing. ATDP dials using pulse dialing.

syntaxerror53@reddit

ACDC - if wanted modem to play some rock music. /s

CrudBert@reddit

Those are modem commands from waaaaaayyyy back. ATZ ( reset), ATDT ( call using dial tone - instead of clicks like a rotary phone). This was called the “AT” command set ( AT meant ATTENTION and was the start of many/most commands). This was originally known as the “Hayes Command Set” as it first came out with the HAYES 1200 baud modem ( I had one, VERY expensive). It was copyright, and other manufacturers that started using it were sued. After some years of ramble ( in which other companies continued to use it anyway) it was settled - presumably $$$ money spread to Hayes to settle the matter. Later on, with modems at 28K and 33K speed ( blinding speed in my opinion) they had the “Extended AT Command Set. This was to handle new features on the newer modems. All in all, they were very, very nice to use!

TinyNiceWolf@reddit

Hayes actually introduced the Hayes AT command set with a 300 baud modem, the "Smartmodem", in 1981. It was $299. The following year they introduced a 1200 baud version for $699.

Hayes was sued by another company which had patented the use of an escape sequence in a modem to switch modes, and in 1983 paid $2 million to license that patent.

Hayes's own patent, for using an escape sequence that included guard time (as the +++ sequence would only be recognized if no other characters were sent for 1 second before and after it), was granted in 1985, and they began trying to license it to other manufacturers using the AT command set for 2% of the retail price of each modem.

Thanks for the clarification. I bought the oh-so-expensive 1200 baud as an upgrade. I’ve told my son for years when he complains that I don’t have the fastest internet they sell, “ in my mind - anything faster than 1200 baud is blindingly fast!” Makes him nuts when I say it, so I love saying it even more. LOL.

The reason the commands all start with AT is due to baud rates. The modem operated over a serial line, and needed to know the baud rate the computer at the other end of the line had been set to use, to make sense of the incoming data and turn it into characters (and to know how to send characters back to the computer).

Rather than have a dial or switch for the user to select the desired baud rate, Hayes decided the modem should auto-detect the baud rate.

But to do that, they needed to have a standard sequence. The modem was programmed to know what bit pattern and timing it would see if the computer sent "AT" at 300 baud or at 1200 or 2400, etc. So it watched the incoming line's raw bits for a pattern it recognized, then switched to the corresponding baud rate.

Oooh, as I recall, AT was a generic attention, and the Z was a modem reset for an acoustic modem. ATDT meant you were going to dial a number which would follow. There was also ATH for hangup. Let's see...

Here you go, the Hayes modem commands.

jimmy66wins@reddit

AT&B2 for the win

Fixes_Computers@reddit

It's been so long since I've had to construct a modem initialization string.

Just this week, someone posted a comment with three pluses in a row. I responded that it took my modem offline. I got a single knowing response which was all I needed

Oh, man, I haven't seen those acronyms in ages! Just when the memory had healed!

The modem connect sound is a wound that will never fully heal.

ATZ
AT&C1&D2&R1&S1&M5
ATDT.....

BBS dialups for the win!

I even ran a BBS on my Atari computer back in the 1980s. It was a wild time.

I ran one on a C64 and then C128 for a while before I moved to an IBM clone. Fun times for sure!

I used to get our home phone number late at night for the BBS, but after a while I was able to talk my parents into getting a second phone line, so I could have it running all the time.

PanoptesIquest@reddit

"AT&T is a modem test command."

AT&T caused many modems to be tested with their shit lines, that's for sure...

SlimJimPoisson@reddit

I was going to say the same thing!

lucky_ducker@reddit

When I was an I.T. manager before retirement, the one guy on my team that knew Cisco CLI was VERY well treated. I certainly didn't know it, and was not inclined to learn it.

Inside-Finish-2128@reddit (OP)

I hear you. In this context though, my personal opinion is that two separate mistakes were made here:

1) the 'default-information' originate command didn't belong everywhere - it only belonged on any router(s) that had direct Internet connections. "I have a way out, and everyone should know it."

2) even if it did belong on a particular router, the 'always' option shouldn't be there. If the underlying default route breaks, it's broken, so why would you share a broken route with the rest of the network? Let all of the other routers know that the route is gone so if anyone hops on any router to troubleshoot, they'll see "OMG the default route is gone. I should go look at the spot where that's supposed to come from!".

Aln76467@reddit

Clueless user here who knows the router as the box between the internet and your computers. Why would there be routers without direct internet connections?

School districts (as one example) often all connect back to a main site (datacenter), and then the datacenter has one (or maybe 2-3 redundant) internet connection(s) for the whole district. This allows the district to have centralized web filter and Internet firewall services. Also the schools would have a mixed need for their traffic: a lot could be going just to the school servers, so the network (“WAN” or wide area network) can be sized for those needs while the main internet connection can be sized for the aggregate need of the whole district.

TheCollegeIntern@reddit

It’s a disaster when it doesn’t work. I had a school that had the high school, elementary school, and the middle school all connected. They onboarded a new switch. Never set the root bridge nor root guard and all sold suffered lots of internet. 3diff techs didn’t pick up on it but I happen to pick up on it and saw the root bridge wasn’t set and asked to set root guard in downstream ports and the sight was up. They rma’d the switches. And did all that jazz just to find it was stp.

Two slogans that stick with me over the years: "Switch where you can, route where you must" and "Switch to fix a traffic problem, route to fix a protocol problem". Both of those would IMHO clearly say one should not switch across a WAN.

Regardless, checking STP should be early on the list of troubleshooting, as is rolling back any obvious changes (like, you know, a new switch).

Penners99@reddit

I learned “route once, switch many” the default workings of a L3 and greater backplane

iwishthisranjunos@reddit

I would say l2 sucks and the suckiness increases by the scale of the broadcast domain. I say route till the end switch only at the start. Routing is easy understanding routing protocols is where often people go wrong ;).

Ok so you'd have a router at each school and they'd all go back to a central one? That makes sense.

TheVojta@reddit

For a more general answer: a router serves as a gate between two (or more) seperate networks to talk to each other. It just so happens that the most common configuration a regular user will encounter is a local network on one side, and the internet on the other.

Damn, so they didn’t leave it to the asbr routers and just assigned it on every router? What the hell?

Right. And they'd only bring us out for specific projects, so no one on our side was ever tasked with looking at broad issues, etc.

faithfulheresy@reddit

It do be like that.

Thanks for the great story.

JaschaE@reddit

I know the cliche of doctors and lawyers being endbosses in tech support, is there a similar one about those responsible for school networks actively sabotaging things (actively, not necessarily)

Know a guy who worked under the citys school network. The 30 or so schools had 10Gbit to each other, and a singilar 1Gbit connection to the outside world. Not to mention the terrible cyberterrorism that made headlines here (in the hastily implemented days of online homeschooling during the lockdown, the shiny new system was victim to a DDOS attack. Who would have thought that on the first monday morning of homeschooling the singular server they had comissioned would have to handle 30schools worth of login requests? Those evil cyberterrorists! Yes, there was a press release ro that end)