When DNS is just a dude
Posted by benjazio_xd@reddit | talesfromtechsupport | View on Reddit | 18 comments
Hey! I've been lurking through this subreddit for more than a decade at this point; I have now become a telecom engineer, and I have some stories to give back to this wonderful place: this is the story of when our nameserver was just a dude.
I had just started working a volunteer position for a local NGO, I was already studying engineering and had been working with these guys for a while, and as the resident young guy that works with computers in a place filled with old people, I just slowly drifted into an IT of sorts; after getting Office running on a couple of laptops and fixing and documenting their heinous email situation, I got some one-on-one time with our librarian:
$librarian: Hey u/benjazio_xd, can you help me with something here?
There was a reference collection of books for internal use, around 30k books in total, managed by this one guy who also cared for the NGOs extensive paper archives, which were around a hundred years old. He's a cool guy who actually turned into a great sidekick for many projects I did while working there, and we remain friends even after I left.
$librarian: You see, we've had this ILS for a while, and I've been told it has an open access catalog so our guys can see what we have and come pick it up, but I've never gotten it to work right, could you take a look at it?
An Integrated Library System (ILS) is a piece of software that tracks pretty much everything inside a modern library: inventory, loans, labeling, shelving, late fees, you name it. They are very niche software but also extremely powerful: they are the beating heart of many libraries, big and small. This one was hosted on a local server in the office itself, and a quick browser check to the local IP address of the server revealed that it did, in fact, have an open access catalog.
$librarian: It's supposed to be on our webpage, but I've got no idea how it works and no one really explained it to me when I got here.
Sure enough, there was a link on our webpage that just returned a blank page every time, and not only that, it seemed to be an internal URL on our webserver, which didn't really make sense considering it was on a different machine halfway around the world.
Nothing in the world would have prepared me for what I saw when I clicked on "Inspect".
$me: So, um, has anyone ever told you anything about this before?
$librarian: the previous girl that had my job told me that the page had to be updated every couple of weeks, and left me a couple of links I had to follow, but she never taught me how to do it and that was like five years ago.
Jesus Christ, this hadn't been working in a long time.
In this blank page was actually an iframe, which pointed to the frontend on our public IP address. This was janky and unnecessary, but what turned it into depravity was one key little detail: we had a dynamic public IP address.
This meant that for years, someone had to connect via FTP to the site every couple of weeks, go to this page, and modify the iframe so the IP address matched to the current one we had. There were no notifications set up either, which meant someone had to notice and tell the librarian that this was going on in order for this to even work, and when they changed librarians no one bothered to write this down, and so that site was just permanently broken: Our dynamic DNS solution was just having a dude update a file on a remote server whenever they noticed the god damn page was down.
$me: This is extremely stupid, how did nobody notice this earlier?
$librarian: you're the first guy here who actually knows enough to care.
My heart sank a little. Apparently this guy had been complaining about this for years but because the dev team for that website was long gone no one had bothered to get someone to look at it. It was an unfortunately common scene in this place, and it was the reason that made me leave it some years later.
$me: Right, this is going to take me about an hour to get everything set up, but I'll get it fixed and running before the day is done.
My solution was just to get a DDNS provider and hook it up to a subdomain of our main site using a CNAME record and just changing the link to the page to the new address. This was fairly low traffic website and just have the server directly respond to requests was fine. I used a small script on the local server running every five minutes to update the IP address to the DDNS provider and that was it: it now just worked on its own.
$librarian: You have no idea how much rage you've removed from my system, let me buy you lunch tomorrow.
We got a static IP a few months later, and I made a friend in the process.
Scipio_Wright@reddit
Yeah, their name is Dennis
Stryker_One@reddit
I wonder how many other networks there are out there in the world where, you REALLY don't want to look under the hood.
onebit@reddit
Dude named sam
jirbu@reddit
Nice job, but ... 20 years ago, it was probably acceptable to connect a dusty, unsupported piece of software to the open internet. Now, that you'll have a static IP, your next support job will probably be to fix all the damage that bots and script kiddies cause by hammering malicious requests to your "fairly low traffic website".
SharkieHaj@reddit
i'm definitely not an expert on this, but can't you just 403 the bots and script kiddies in some way?
henke37@reddit
The trick is figuring out which requests come from real users and which are internet noise.
kotenok2000@reddit
It might be beneficial to also drop all connections to ports other than http/https.
meitemark@reddit
I have seen a system where every port was open, but unless it was http/https each port was limited to 1 byte/s. Pretty sure it was just to showoff what their router/firewall could do, but systems that are "open" but incredibly slow or gives timeouts all the time is great ways to trap both bots and script kiddies.
RedditWhileIWerk@reddit
tarpit, nice!
Jonathan_the_Nerd@reddit
402 would be more profitable.
benjazio_xd@reddit (OP)
Oh, absolutely. This was years ago, but we actually overhauled the entire network when we got the static IP. We send our traffic through cloudflare proxy and then through a hardware firewall on site.
harrywwc@reddit
as we say, "it's always DNS" ;)
Jonathan_the_Nerd@reddit
It's not DNS.
There's no way it's DNS.
It was DNS.
MrDeeJayy@reddit
I say this so often and yet the other day I got hit with one of these, a client laptop would work on ethernet and wouldnt work on wifi, turned out that months ago while the DHCP server was acting up and not advertising the correct DNS server, I manually assigned the DNS server, then that DNS server migrated to new hardware and to a new IP, and I never fixed it.
LordGopu@reddit
Dude Naming System
Cmd_Line_Commando@reddit
Heh reminds me of Sneaker Net.
Ich_mag_Kartoffeln@reddit
Frisbee Net.
4rd_Prefect@reddit
Awesome fix & ++1 for making a friend 😁