I need to create a failover DNS server on a Rocky Linux 10 KVM for my university.

Posted by retired_nihilist@reddit | linuxadmin | View on Reddit | 23 comments

Hello!

I help manage the network services for my university's faculty. We're trying to align with tier 2 uptime standards, and my professor asked me to set up a "mirror" DNS server.

Currently, we have a primary DNS server with a public IP, and I was given a separate phisical server with Rocky Linux 10 Minimal where I have to create a KVM virtual machine on it and configure it as the secondary DNS so that if the primary goes offline, this new VM handles the resolution without downtime.

I've set up basic DNS servers before as a lab experiment, but I haven't tackled a proper production setup yet.

A few things I'm trying to figure out:

Is the set up as simple as in a lab environment or are there any concepts that I'm missing?
How can I keep the secondary server updated in real time? Is there an enterprise-level approach?
I assume I need to set up a network bridge on the Rocky host so the VM gets its own IP on the same subnet (I have done this in the experiment I mencioned). Is this the standard practice for DNS VMs?
Are there any common pitfalls when setting this up in a production environment?

I've been searching for tutorials, but most just cover basic single-node setups. Any pointers to good documentation or advice on how you'd architect this would be awesome. Thanks!

[-]

rankinrez@reddit

How many people use this network?

You seem a little out of your depth tbh. But basically copy 1:1 what you did with the first and assign the second dns server IP as well as the first to clients with DHCP / however you assign their IPs.

If you’re talking authdns it’s slightly different but kinda the same.

[-]

bigbluebus73@reddit

Does your central IT not do DNS? Just use theirs?

There is Zero reason for departments to run their own basic services unless it is empire building on the part of the dept head.

[-]

retired_nihilist@reddit (OP)

honestly, I dunno why do we have this. uni has a central IT department, however, our faculty (CS Faculty) has its own infra, its own databases, web servers, DNS, files servers, etc.

my professor told me that he was hired because the last one had everything on AWS (dunno if it's true) and it was really expensive for the fac to pay it, so he showed that it was cheaper when running BM servers.

I find interesting the way he manages all the services, however, I think it's antiquated and I'm really curious about how a well-established enterprise manages its services.

[-]

_piotrm@reddit

Ok, why HA hasn't been mentioned? Wouldn't it be easier? https://docs.rockylinux.org/10/books/web_services/07-high-availability/

[-]

mad_redhatter@reddit

You want a secondary. Failovers with DNS aren't really a thing. The client tries the secondary if it cannot reach the primary.

Look up also-notify in your primary's config if you're running BIND. It will help you to sync record changes from the primary onto the secondary.

[-]

retired_nihilist@reddit (OP)

Thanks! I saw something about that directive, so I will keep that in mind. :)

[-]

chock-a-block@reddit

In the typical environment where the dhcp server is the one sending the name servers, You set up “failover” by adding a second dns server to the settings the dhcp server sends to clients.

[-]

TabTwo0711@reddit

Please don’t, hier someone who knows his stuff

[-]

retired_nihilist@reddit (OP)

When will I learn then? I want to become a sysadmin/devops engineer/sre engineer, and I was really excited when my professor tasked me this \^\^

[-]

TabTwo0711@reddit

You are going to set up a potential DOS amplifier in a high bandwidth environment. Running a public DNS server is a job that requires especially one thing: years of knowledge that can’t be replaced by a question on Reddit and some googling. Get professional help. They may also like the idea of teaching you stuff. First thing you Doris go to the local bibliotheca and get a copy of „DNS and Bind“

[-]

retired_nihilist@reddit (OP)

I wil talk with my professor and take a look to the book, thank you for the tips

[-]

SuperQue@reddit

So, DNS doesn't really have "failover" or "mirror". It's just not in the DNS design.

It sounds like what you're looking for is a "secondary" DNS server. These typically use AXFR to replicate the zones from an authoritative server.

Is the set up as simple as in a lab environment or are there any concepts that I'm missing?

Absolutely, easy to do in a lab environment. You just need to setup to nodes and configure DNS servers on them.

I recommend using CoreDNS. It's easy to use, flexible, and fast.

How can I keep the secondary server updated in real time?

Use AXFR secondary configuration.

I assume I need to set up a network bridge on

Yes, but why bother with a VM? If you have a bare metal server, just use it as-is. Seems like an extra complication for no reason.

Are there any common pitfalls when setting this up in a production environment?

You need good monitoring.

The good news is, CoreDNS has this built in.

[-]

Ludwig234@reddit

You can setup anycast for DNS though.

That way you can loadbalance and achieve HA using a single IP.

This is for example how the root servers are setup.

[-]

SuperQue@reddit

Sure, but I didn't want to get into that with a newbie. There's also OSPF/HSRP/CARP methods without having to go to Anycast.

[-]

retired_nihilist@reddit (OP)

Thanks a lot for the useful information! About the VM thing, I honestly don't know why my professor tasked me to create the DNS setup inside it; in fact, all the services are inside a VM, and each VM is hosted on a bare metal server :l

I will take a look to the monitoring point. Again, thank you! \^\^

[-]

SuperQue@reddit

If it's just for practice, a VM on the same machine is fine. But for "production" you usually want separate hardware for primary/secondary.

Using VMs is common if there are a bunch of different services on the same hardware.

I would recommend checking out the Cloud Native Computing Foundation slack chat.

[-]

michaelpaoli@reddit

There are essentially two parts to this:

First, the DNS servers proper. DNS is generally set up to be quite highly available and redundant - that's the typical infrastructure in general. However, and given server can fail, but DNS - generally well have the redundancy, e.g. multiple (e.g. generally highly recommended 3 or more) authoritative servers, etc. And, those can be set up as primary/secondary(/ies), or as multiple primaries. Either is generally fine, and much DNS software, it's not too hard to change a secondary into a primary if ever there's need or desire to do so. So, that's most but not all of it.

The bit DNS servers generally don't do for you, is HA on the individual IPs of the DNS servers. So, do that with something else to cover HA for the IPs. So, have your DNS servers on VIPs, a VIP goes down or becomes inaccessible, well, HA activates that VIP on another server, and then the IP continues to be fully functional - it' just now on a different host, that's all. And well configured, the HA should also be capable of well failing it back, when the host that had issue is able to pick up and continue doing the serving again.

So, that's pretty much it - good base redundancy and replication, etc. with the DNS servers, and then a HA layer to make the server IPs HA, so the service is available most all the time on all the relevant IPs. That should pretty easily get one 5 nines or better of availability, at least with proper management (e.g. don't take 'am all down at the same time for maintenance, or likewise any infrastructure their services depend upon).

[-]

retired_nihilist@reddit (OP)

Thank you for your reply! I'm learning a lot because of this task and all the comments<3

[-]

dodexahedron@reddit

Pretty quick and easy if you want to use a container or just run a service on the physical host.

Grab a BIND container image or install BIND on the server.
Configure a zone for youruniversity.edu and set type secondary; and primaries { ip.of.mas.ter; }
Allow axfr and ixfr to that secondary from the primary.
Set notify to that secondary from the primary (causes updates to trigger the secondary to refresh it from the primary)
Be sure there is an A/AAAA record for the secondary in the zone on the primary, and an NS record pointing to it.
Create a glue A and NS record at the registrar with the same definitions as step 5 (or else it is only visible to systems that have already found the primary).
Enjoy having a primary and secondary.

Or...

Go set up an account at any one of the many public secondary dns providers out there.
Follow steps 3-7 from the above list, using that public service instead.

For example, dns.he.net is a free option and is run by one of the largest tier 1 transit providers in the world, and provides you a geographically-diverse, high-uptime, and high-capacity set of secondaries.

Offloading to a public service like that takes all that DNS traffic off your network and keeps you from running an9ther internet-facing system that really should not be exposed without understanding it in depth, for security and other operational reasons.

You can even run a "private primary" where your primary is not listed with an NS record in the zone or at the registrar, but all the secondaries are configured to point to it as the primary. Then the primary never receives queries, never is exposed to hostile internet clients, and is ONLY a source of the authoritative zone for the secondaries to all serve up to DNS clients. There is no reason not to run a private primary these days. Even if you need to support internet-facing dynamic DNS updates, secondaries can handle that if configured properly.

[-]

retired_nihilist@reddit (OP)

Pretty well explained, thanks for the info, I'm already searching in how to prevent DNS amplification and how to set the service up in the proper manner.

[-]

dougs1965@reddit

A possible reason for using a VM is so that when the bare-metal server breaks or is needed elsewhere, you can just move the VM to another BM server, it's much easier than moving that service from one physical machine to another. You can replace all your bare metal in rotation, moving VMs from old servers onto new ones as they become available and there's minimum fuss and (particularly helpful in your environment) minimum downtime.

[-]

retired_nihilist@reddit (OP)

You're absolutely right! I hadn't thought that, but it makes totally sense, thanks!

[-]

therouterguy@reddit

Every piece of dns server can do this. I would set this up by creating a bunch of secondary servers behind a loadbalancer. These secondaries are being kept up to date by a primary via axfr/ixfr/notify process.