Why windows admins don't make good linux admins

Posted by zeckz@reddit | talesfromtechsupport | View on Reddit | 33 comments

Worked a few years ago at a one-stop-shop level 1, 2, and 3 linux tech support role, dealing with lots of government clients with a wide variety of issues. One particular client had an issue pop up where all outgoing emails from their domain were being rewritten, for example: dude@company.com was rewritten to dude@www.company.com. This caused havoc for any email recipients, as they couldn’t reply - the email server did not receive the www addressed emails. The client's first request - restart the server. We said no, because you don't reboot Linux servers at the first sign of trouble, it's not Windows! After a lengthy investigation, the issue was identified as a hosts file entry on their email server with the following: > /etc/hosts: 127.0.0.1 www.company.com company.com So what this entry in the hosts file means is that whenever a DNS lookup was performed for the domain *company.com*, such as on outgoing emails, the response was the *www.company.com* domain instead, because hosts file entries override any external DNS response. This issue is easily fixed by removing the hosts entry, but the reason the customer had added this was because the server had no DNS server configuration listed in resolve.conf, or more to the point it had been removed (given the file update time on resolv.conf), meaning that DNS lookups didn’t work at all and no email was currently sending. When the client was queried to see whether we could (re)enable DNS config in resolv.conf to fix this, the client responded with the following: > **Client** (formatting preserved verbatim from email): “why it’s now required since it hasn’t been config in the last 5 years? … it makes sense that the DNS is not enabled as it’s an internet facing server and to have DNS server in the config is actually a risk as when the system gets compromised, the malware will easily be able to get to its control centre.” The server had been configured for DNS lookups for the last 5 years, the client was flat out lying. Linux timestamps don’t lie. Also it's internet facing because ya know... it sends emails... on the internet! Also the OS in question was Red Hat Linux, and I have never heard of “malware” getting to its “control centre”. We facepalmed a lot that day, and this wasn't the end of the absurdity from the ~~Windows~~ Linux Admin client.

33 Comments

[-]

gex80@reddit

OP takes a dig as windows admins for a problem that had 0 to do with windows and shows their misunderstanding of how host files work. Also I’ve had instances where rebooting a Linux Machine fixes the issue.

ontheroadtonull@reddit

\> “malware” getting to its “control centre” Referring to ransomware reaching out to a command and control server?

deNederlander@reddit

Yes, absolutely. Bad OP

I can't read the post again since it was deleted but I think my initial interpretation was that OP inferred that the client believed that "control centre" was some facility built into Linux. Let me rephrase: OP thought that the client told him that something they were doing would prevent malware from gaining control of something important in the operating system. Probably time to just drop it and move on to the next awful thing.

whatever462672@reddit

Yea, this.

pandab34r@reddit

It sounds like the real underlying issue is improper DNS config which, with enough effort and the right skillset, could be accomplished on any OS

That is not how any of that works. That entry in etc/hosts resolves company.com and www.company.com to localhost. It doesn't advertise them to anywhere other than localhost. You are thinking backward.

hallibax@reddit

I still don't get what this has to do with being a Windows admin vs being a Linux admin?.. Rebooting something that doesn't work shouldn't be the first solution for problems on any OS, but sometimes it's a good solution for problems on all OS. All OS's are complex beasts..

LucasPisaCielo@reddit

"The client's first request - restart the server." In Linux, you would try to reset the email service first or many other solutions before thinking on rebooting the server. Client's request was an indication that in all likelihood he was a Windows user.

Phrewfuf@reddit

But sadly rebooting is often the solution. Basically, programmers make a lot of assumptions when designing their software. Specifically assumptions about the state of the process and the host. Some are better at it than others. But the complexity is far beyond an individuals reach of comprehension, so things constantly drift off. Sometimes they drift off far enough for an assumed state to not longer be true, resulting in unexpected behavior. Rebooting resets the entire host to a known working state. Sometimes things are so critical that you first reboot to restore the service and then investigate what caused the outage. Specifically: see if it happens again and what might cause it.

ih-shah-may-ehl@reddit

And honestly you should have the redundancy in place that no individual server is critical. If 1 server is so critical you cannot reboot it, you're running amateur hour and you're 1 failed component away from disaster.

Caeremonia@reddit

OP, you need to just go ahead and delete this post now and save yourself from more shame and embarrassment.

rhavenn@reddit

That’s not what that hosts entry means. It just means that www.company.com and company.com will resolve to 127.0.0.1 aka:local host and skip a DNS lookup. You can put any number of host names on the 1 line. Your problem was more likely that a lot of Linux mail servers will send out email using the host name and a lot of linuxes will use the 127.0.0.1 / local host entry for their host name if one isn’t defined explicitly elsewhere. So, the system was just sending mail out as user @defaulthostname. You could have re-configured the mail server too to use whatever domain you wanted. But lol..no dns lookups. Server was never updated then either unless they had hosts entries for the update servers.

Thanks for saving me the typing on this one. I concur with your diagnosis of the hostname not being explicitly defined. I love the condescending tone of OP while simultaneously not understanding something basic like how the hosts file is formatted and how it fits into a DNS lookup.

This is probably the issue. This hostname should be set in /etc/hostname or where ever your distro recommends it. Otherwise you will find your servers with very creative ways to infer the hostnames themself, like using the host file.. Also, in general, an email server should not send emails with the hostname as the domain. That should also be configured explicitly..

TrippTrappTrinn@reddit

Title is wrong. No proper Windows admin would set up a server without proper DNS settings. Pure incompetence.

aard_fi@reddit

> Also without DNS, how could it look up the MX records for the external recipients MX records are optional, if they don't exist fallback is to the A (or AAAA) records - which it got via hosts.

Still needs DNS for that

Mastacheata@reddit

The hosts file can resolve its own hostname, but not for example where to find the Mailserver for eric@gmail.com. That's what's meant when referring to MX records for external recipients.

It can resolve arbitrary names to the equivalent of A or AAAA records. If the mail server runs on the same host everything is fine, due to the fallback mechanism for missing MX I described above.

Yeah, but they didn't specifically block the Mailserver from looking up mx records, they disabled DNS resolution on that server completely is how I've read it.

FapNowPayLater@reddit

MS server used to let you do this. Pre 2003.

Atacx@reddit

The server had DNS, he said this after the quotation. Just the one entry from the hosts file gets overridden

muusandskwirrel@reddit

Ah yea, malware whose Cnc is local host.

ShelLuser42@reddit

I can answer #2: it's not uncommon to set up a mailserver with a dedicated forwarder host. In other words: it doesn't use MX lookups but simply sends *all* its mail to its "uplink". And I can easily see why someone would prefer the mail part to be handled on Linux.

kagato87@reddit

Even for a windows admin, you don't just reboot the servers. Desktops yes, servers no. Don't be too dismissive of a windows admin. Chances are we've seen things you haven't (especially since the kernel isn't isolated as well) and are cursing the failings of a dominant technology. This windows admin will tell you that you've missed a second, larger issue: A bad hosts entry should not have created that problem. That's a bug report for whatever did the rewrite.

Not sure how you meant your 2nd part. But I'm a windows system software dev, currently working as an admin. You can read the details in 'windows internals 6th revision' where a lot of things are explained in full detail. Basically what you think of as 'the kernel really isn't. Not since windows 2016. Basically the true kernel is a very thin hypervisor that only runs a few specific Microsoft drivers, the lsa, input and a few other things. The 'full' windows kernel runs on top of that. Not even kernel drivers or services can touch the core windows security boundaries anymore.

xgnarf@reddit

I would argue that it's not that windows sysadmins are bad at linux, but more windows allows terrible sysadmins to hide their incompetence longer. If one is a good sysadmin the OS is irrelevant as they would be able to follow proper documentation and best practices regardless of linux or windows. Windows is just easier to see and identify problems

xzer@reddit

>I have never heard of “malware” getting to its “control centre” I'm pretty sure in the last 10-15 years malware on Linux based systems has really bloomed. It's a pretty funny approach though, since, if I was was going to make malware I wouldn't rely on DNS to get an outbound connection lol.

DMercenary@reddit

The opposite admin sounds like they've read X article on internet and decided to "fix" the hole they have on their server.

ittimjones@reddit

This sounds like "I don't know what I'm talking about, but I'm going to act like I do so nobody knows." Fortunately for our line of work, we can all easily identify and chastise these individuals. (I do it alot, and I am feared. Lol)

Geminii27@reddit

There's definitely a difference between a general systems admin and a one-focus admin.

blackAngel88@reddit

> https://www.quora.com/Why-did-the-phrase-Who-is-buried-in-Grants-Tomb-become-so-famous Well... this is not really limited to Windows tbh. I wouldn't exactly just randomly restart any prod server because 1 thing doesn't work... But I've also had a case where a query on a postgres server would work the first time and every other time on the same connection it would throw an error. It took us way too long to just think of restarting the postgres service and the problem was fixed...

Reply to Post

33 Comments