It's always DNS, even if it absolutely shouldn't be.
Posted by scribeawoken@reddit | talesfromtechsupport | View on Reddit | 40 comments
I used to do call center tech support for a major American ISP up until I quit back in August after getting fed up with how much it felt like our policies were designed more to find excuses to avoid sending out a technician than to actually identify and find the solution for an issue.
I was one of the few people there who actually had a technical background, and during my time there, I had a knack for identifying a lot of more in-depth issues that very few people around the call center ever would've picked up on - either because it wasn't in our troubleshooting script, or it was so many layers deep in the script that most people would give up and dismiss the issue out of hand as out of our scope of support. This is a case of the latter.
This occurred about a month after I finished training, and I was working second shift at the time. Relatively early in my workday (around 4 PM I wanna say), a call came in from a supervisor in our cable TV support department. He had taken an escalation from a customer who called in about an issue with his internet service, and it was clearly more in-depth than the basic troubleshooting his department handled. I let him know that I could take the call, and he transferred the caller on over to me.
After taking the steps to verify the caller's identity, I asked him some info on the situation. The gist was that he'd been unable to get online on his Mac, and he just had a technician out. This technician basically just showed up, dismissed it as an issue with the customer's third-party router, and left. This caller wasn't having it, and called in stark raving mad and demanded to speak with a supervisor about it.
I managed to de-escalate him a bit by assuring him that I was going to do some more in-depth troubleshooting, just to do my due diligence and identify if this issue truly was with his router or if it was unrelated. Luckily, the caller had already bypassed his router and hooked his Mac directly up to his cable modem, which made things a lot easier, since we just had to verify whether it was an odd fault with the modem that the field tech had ignored or an issue with the customer's Mac.
Since the basic steps of power cycling the modem and rebooting the desktop didn't do anything, and Safari would just stop responding whenever he tried to go to a webpage, I asked him to open up the terminal and ping 8.8.8.8, which went through just fine. I then had him try pinging google.com. No dice.
Afterwards I had him check the network settings to see what the DNS settings were. It was set to manual configuration, with a single IP address listed:
127.0.0.1
Somehow, for some ungodly reason, this man's computer was configured to use its own loopback interface as its sole DNS server. I do not know how this happened. I'm not sure if I want to know how this happened, but it happened.
From there, it was a quick case of advising him that he could either use something like Google DNS or Cloudflare DNS, or he could set it to be configured automatically via DHCP, and he was on his way.
The funny thing is, he assumed that I was a supervisor the entire time. It wasn't until the very end of the call that I told him that I was only a month out of training.
Thetechguru_net@reddit
I have spent my entire (30+ year) career in positions related to technical customer service. It never ceases to amaze me how bad most people are at troubleshooting even when it is a primary part of their job description. I have tried to teach it, and failed miserably with every attempt, and otherwise I am a good trainer. It seems to be an innate skill you either have or you don't have. You clearly have it, so good for you, it can take you very far if you want to stay in the field.
NotYourNanny@reddit
Proper troubleshooting requires an analytical mindset that is very difficult to learn as an adult. And it isn't taught in schools any more, and hasn't been for a long time.
crimsonpowder@reddit
It comes down how much curiosity and tinkering someone did with something, anything when they were younger. Most people just go through life following the script society gives them: go to this class, participate in that group, attend this activity, etc and so it shouldn't be shocking that they need frameworks and guardrails all around them for the rest of their lives.
One could also argue that it's more energy efficient to not experiment all the time and just follow the "life script".
Amber9572@reddit
It also has to do with how much people were scolded as children for “coloring outside the lines,” so to speak
TheCollegeIntern@reddit
Most people especially in college are simply going to class then they graduate and wonder why they cannot find a job.
Responsible-End7361@reddit
It was taught in achools? When? I'm Gen X and it wasn't taught in schools during my school years, nor the 30 year period that I have had one kid or another in school since...
TheCollegeIntern@reddit
Technical problem solving wasn't taught at my school, however my school did teach critical thinking as a course elective and it was always the favorite subject I had in secondary school, middle school and high school. I would always sign up for that Class.
Diaggen@reddit
I was taught how to problem solve in HS. There wasn't a formal analytical thinking class. However, I did take machine shop, wood shop, and drafting. All taught by a WW2 vet who served as a machinist aboard ship in the Pacific theater.
Anyone who couldn't think their way through how to solve a problem by the end of a year with him was just not trying. I've trained IT and support staff and worked in the field for close to 30 years. The most skilled and easiest to train almost always had taken a vo-tech class or had a hobby or prior job that involved some level of creating or fixing actual physical things with their hands. They just weren't afraid of digging in and possibly breaking shit and then figuring out how to fix it.
Geeky-resonance@reddit
Several years ago, some researchers decided to look into differences in education between the generation that pioneered the space program and more recent generations.
IIRC the conclusion was that hands-on skill classes like “shop” were the difference that helped students develop much better problem solving skills. Classes like this were largely eliminated as fewer graduates entered the workforce immediately and US secondary schools focused more on college preparation.
Surprise, surprise: educating the whole person, including life skills, instead of focusing solely upon testable academic benchmarks produces well rounded, functional, productive adults. Wish that had been the approach during my K-12 years.
derKestrel@reddit
With this approach one also develops a feeling after a while for what should be easy to fix if you break it, so you can go for it and "diagnose destructively", and conversely what you should not even try to touch because woe to you (and possibly your weekend).
quadralien@reddit
Even then, it's hard with disobedient and deceptive remote hands.
NotYourNanny@reddit
Very true. But it's a lot harder if you don't know where to start.
Geeky-resonance@reddit
That’s a fancy way to say that users lie, yes?
scribeawoken@reddit (OP)
honestly from what I can tell, whether someone learned that analytical mindset in school is less a matter of when and more a matter of where and who - teaching styles and curricula can vary wildly between schools and even between teachers within the same school.
also honestly tbh I think a well-written script can be a good crutch to deal with the most common issues, but "well-written" and "crutch" are both key things to keep in mind there - if you can cover the most common issues, that's great, but also you need to give people the leeway to break from the script if something outside those issues, or at least have someone they can escalate to that isn't bound by it.
More than half of the technical customer service nightmare stories I've heard from the customer side of things boiled down to the script being too narrow in what issues it covered and the company clearly not having any procedures in place for issues that the script didn't cover, instead just treating the issues that the script covered as the only ones that could ever occur.
scribeawoken@reddit (OP)
Yeah. I actually remembered this story while working on a cover letter for a help desk job I'm applying for.
For me, a lot of it was just... getting a feel for the procedures we were supposed to follow, taking the time to understand the actual underlying logic behind them... and also understanding what situations could lead to that logic falling apart, and how to improvise when that happens. I was the guy who knew to look for horses first when I heard hoofbeats, but not to blindly assume that it can't be zebras - and over time also developed a knack for quickly identifying what hoofbeat patterns did in fact indicate there were zebras afoot.
And then management got more and more strict about us just looking for horses, and told us that if we find a zebra, we should reach out to L2 support. Only to have to deal with an L2 support rep who basically tried to gaslight me and insisted that the zebra I found was in fact a horse, and that I just didn't know what I was doing.
Originally I was going to wait until I had something else lined up and put in my two weeks, but between that and getting hit with a return to office mandate and not having reliable transportation, I just said fuck it and quit on the spot.
Frobbotzim@reddit
Oh man, I love the horses and zebras simile. That was so much of what I tried to pass on to my teammates: know the obvious, look for the obvious, but don't assume the obvious when the facts don't quite line up, keep digging for the details and get familiar with the edge cases because they're everywhere. Worked with a lot of CCNP's that needed this kind of guidance repeatedly.
Good luck nailing that help desk gig--you got this, and just reading your post & comments, I feel like that's aiming low (none of my bizness and there's a lot of great reasons to grab tier 1 with a good org and start shining like the sun, just saying, I wish that I'd had a few more peers with what you've got in my division before I packed it in a few months back (yep, mgmt broke me too)).
scribeawoken@reddit (OP)
Application for the help desk gig was seemingly rejected out of hand since I'm not in the same city - even though I mentioned in my cover letter that I am able and willing to relocate (I live just outside the greater metro area and have family in the city that I can move in with). Oh well.
Frobbotzim@reddit
Ugh, I am very sorry to hear that. Keeping fingers crossed for you, good Internet stranger, that you're able to pick up something that allows you to spend every day with all the interesting technical problems and a minimal amount of the daily layer 8 shenanigans that strip the fun away.
scribeawoken@reddit (OP)
here's hoping. honestly I'm also kinda leaning towards my backup plan of leveraging my customer service experience for an administrative assistant job, especially since I saw a listing for one that's, like half a mile away from the relatives I plan on moving in with.
either way, hopefully I'll be done with dealing with an oddly high number of people who seemingly don't know basic shapes and colors.
highwarlok@reddit
Trouble shooting is related to critical thinking and it is not taught.
TheCollegeIntern@reddit
Critical thinking was taught how throughout my grade school experience and it was one of my favorite subjects. You absolute can teach critical thinking but I guess by a certain age it might be less effective.
Slackingatmyjob@reddit
Critical thinking is related to thinking, which isn't taught nowadays until post-mandatory education
Elementary and high school levels it's all about rote memorization
androshalforc1@reddit
Companies don’t want people who can think they cost money.
They want people who can follow a script
Step 1) have you tried turning it off and back on?
Step 2) ~~escalate to tier 2~~ tier 2 is redundant due to having tier 3, please see step 3
Step 3 ) ~~escalate to tier 3~~ tier 3 has been deemed too costly Asha been removed, please see step 1
Brendoshi@reddit
Oh, this is really good to hear. I thought I was doing something wrong with my attempts
capn_kwick@reddit
Asca wild guess, software was installed that, as part of the install, would reset the DNS settings.
SeanBZA@reddit
Or unistalled some third party firewall or antivirus, that included in it's own suite a DNS resolver, thus the need to send all connections to localhost, as that would have had the suite handle it correctly.
mrkorb@reddit
Ugh, freaking DNS. Over a decade ago I had this printer at work that I could not for the life of me get to work over the LAN for nearly 2 weeks. I had my boss breathing down my neck over it. It was only when I filled in the DNS server field with good old 8.8.8.8 that this locally networked printer was finally able to connect with the locally networked PCs. In hindsight, yeah, okay, I get it, but it just felt like such an unnecessary requirement for a device that was never going to be fielding actual internet traffic that it didn’t even occur to me that I shouldn’t just leave that network configuration field blank.
joule_thief@reddit
Yep, having been in a similar role it's amazing how many folks couldn't do a simple ipconfig /all or ifconfig -a to see what DNS was set to.
Of course, I also knew how to remote into the modem and run a ping from it to verify the modem was working. As I was leaving tools were being rolled out to do that from a web interface. I'd like to say I believe it would get better but I know that won't be the case.
Being more knowledgeable just meant I got more work to do. In that situation, I don't recommend it. I was the go-to person for voice and was on back-to-back escalation calls all day fixing stuff that other people messed up. Fixing port-ins was one of the reasons I finally started looking for another job.
Astrolltatur@reddit
Thx for post it reminded me I needed to call my ISP for my internet problem.
I know a bit about this stuff but honestly the only thing I troubleshoot at home is port forwarding and the easy stuff.
I was told that my router was likely faulty after reporting I get those random internet disconnects on random times I replace router a week ago last night I had 2 disconnects during the night so now I'm so out of my comfort zone since it is most likely my fiber box so ye hope some tech will fix it it's annoying little bit of trouble :D
katmndoo@reddit
Well done!
Maybe he installed some sort of local dns server then forgot it then borked it. Unlikely unless he had been going down some privacy rabbit hole or something.
scribeawoken@reddit (OP)
That's probably the most likely answer. iirc it started right after he upgraded to what was at the time the newest version of macOS, so that could be a factor.
Loko8765@reddit
He probably had some filtering “security” DNS server that either got uninstalled by the upgrade or that he uninstalled… maybe because he had problems accessing the Internet for some other reason that was later fixed!
scribeawoken@reddit (OP)
That could do it. Could've been that the version he was on just wasn't compatible with the newer version of MacOS he upgraded to (Apple doesn't much care about maintaining backwards compatibility with older applications), so it stopped running after the system update.
megared17@reddit
My workstation is set to use 127.0 0.1 and it works perfectly
mgedmin@reddit
You have a local DNS resolver running.
megared17@reddit
Haha.. you caught me. Was hoping to get some confused replies telling me it wasn't possible :)
meitemark@reddit
Well, since the leased line is down, you can't reach the DNS servers, so yes, it is in fact a DNS problem.
"Damned if you do, Damned if you don't."
"DNS if you do, DNS if you don't"
nighthawke75@reddit
Back in the XP days, sometimes Windows would get a wild hair and put 127.0.0.1 in DNS for no obvious reason during the CHAP handshake. That was one of the first places we'd shop if someone was able to log on, but no internet. And there it would be.
Traveling-Techie@reddit
I would be interested to know: did it ever work? If so, what happened just before it stopped working?
murlock42@reddit
More likely a local resolver installed by a malware or an antivirus