First time migrating “primary” DC

Posted by HarlanGames@reddit | sysadmin | View on Reddit | 37 comments

I’m assuming it’s normal, but wow that was stressful everything seems to be working fine post operation. Just glad I don’t have to do it again for a couple years.

We pushed it off so long, it finally no more 2012r2 DC’s.

[-]

jtheh@reddit

well

add new DC
test
seize FSMO roles
test
make sure direct pointers to old DC IP or DNS are changed
make sure no other essential services are running only on your old DC
shut down old DC for scream test
if no issues after several weeks - and before your tombstone lifetime expires, boot it up again, demote and remove
if only 2016+ DCs remain, consider to raise DC and forest function level to 2016
cheers

[-]

thelastquesadilla@reddit

Bad Advice: You never power on a DC that's had it's roles seized. If the DC is on, you migrate the roles. Seizing a role is only a last resort option if the DC is never coming back.

[-]

jtheh@reddit

you are correct, "seize" was the wrong word, I changed it to transfer

[-]

extremetempz@reddit

I recently did the same thing in 2 forests 5 domains, ended up migrating 18 domain controllers from Server 2012 R2 to Server 2022

[-]

extremetempz@reddit

We have 2 datacentres that house DCs, then 2 remote offices that have 2 DCs each (2 different domains )with 5 domains and 2 forests it adds up even if you only do 1 in each location

[-]

Yeah, We do 2 at each site. We've got 4 sites with 2 domains each (16 DC's). If you're licensed for datacenter you might as well spin them up. Don't want things going down when you need to apply updates.

[-]

Physics_Prop@reddit

I never understood people running so many DCs for such a small environment.

We had 70 sites and 15K users, only 3 DCs. Firewall would run a local DNS service to forward the AD zone. Running DCs at each site would be an unacceptable level of risk, we couldn't control each site like we do our datacenters.

[-]

extremetempz@reddit

Well we have 300 sites and 13k users, 4 locations have DCs I would say this is bare minimum for us, if DNS goes down we effectively kill the network

[-]

Physics_Prop@reddit

That's the key, don't tie the dns you give out via DHCP to AD.

Forward your AD zone from the DNS service on your FW to your DC(s)

Few advantages:

1) You get HA without having to give our 2 IPs via DHCP, so your clients can't bind to the wrong DC and do DNS over a WAN VPN

2) Easier to maintain, don't change DHCP and wait 8 hours, change the IPs in your FW if you make a new DC in a new IP.

3) If the worst happens and the DCs go down, the Internet is still live. Only the zone for your AD is unresponsive.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClFcCAK

[-]

RichardJimmy48@reddit

The disadvantage with this approach is it can make using AD Sites and Services more difficult (but not impossible) to get working properly. If there's any kind of NAT/tunneling in between the workstations and the domain controllers, you'll need to make sure whatever subnet the DC sees as the source address on the request is in the site the workstation should be in. In your setup, you'll also need to make sure that's true for the firewall that's doing DNS forwarding. The DC will need to see that firewall as being in the correct site for the workstations it's serving. Not the end of the world, but it is something that will need to be precisely configured and can be a hassle if your network designer and your AD administrator aren't the same person.

AD Sites can matter if your office locations have things like local file servers and you're using DFS-N to have users get referred to their closest file server, or if you want to automatically add printers to a workstation based on location. If all of your remote sites are bare bones with no local assets, then it won't really matter.

[-]

Physics_Prop@reddit

Why would you want to NAT in between sites?

DFS namespaces and any other services I've seen all work flawlessly behind a DNS forwarder, DNS is DNS, unless you are doing something really funky like EDNS or split horizon, none of these services care about how an answer gets resolved.

Sites & Services was built for a time when we measured Internet speed in kbps, assuming you have a stable network, a few sub-optimal cross country replications are irrelevant.

[-]

RichardJimmy48@reddit

Why would you want to NAT in between sites?

I dunno, maybe you have more than one tunnel and don't want any kind of asymmetric routing to happen and SNAT things as they leave the firewall. People do it all the time. It's extremely common and I'm surprised that you're surprised by the notion.

DNS is DNS

DNS is DNS, but Active Directory is also Active Directory, and things like site detection and service discovery happen via DNS, and the domain controllers make decisions on how to respond to those DNS requests based on the source IP address of the request. If you get it wrong, suddenly your user in New York is printing to printers in Boston and their home directory is mapped to a file server in Dublin. You can say DNS is DNS, but you're not going to find a lot of seasoned AD admins who want anything to do with a network where there's a DNS layer in between the workstations and the domain controllers. When you get everything exactly perfect it will work fine, but every change from there on out is going to be fraught with peril.

[-]

Physics_Prop@reddit

Run a routing protocol between your sites, lets you have as many tunnels or EVPN or dark fiber, whatever between sites. NAT between sites is ridiculous and doesn't scale.

Service discovery happens through resource records SRV records, which don't care if you get forwarded. And yes, a lot of seasoned admins don't understand DNA because they have only ever clickopsed Microsoft products and don't understand the underlying implications of what they are doing and why.

[-]

RichardJimmy48@reddit

Run a routing protocol between your sites, lets you have as many tunnels or EVPN or dark fiber, whatever between sites. NAT between sites is ridiculous and doesn't scale.

You can run whatever routing protocol you want, RIP, BGP, OSPF, EIGRP. None of them guarantee a packet will return to the same firewall from which it came.

Service discovery happens through resource records, SRV records, which don't care if you get forwarded.

I suggest you spend some time learning how a lot of Active Directory internals work, because you seem to be lacking some critical information. AD fundamentally relies on DNS records, and features like Sites and Services work based on the source IP of the request. Should they work that way? Probably not. Do they? Yes.

[-]

jrichey98@reddit

Running DCs at each site would be an unacceptable level of risk.

No. It's the same profile for your endpoint protection, and since they replicate with each other. If you can compromise any then you can compromise one, so there's no difference in threat.

... we couldn't control each site like we do our datacenters.

Why? You couldn't remote into your off sites? Not like you'd ever really need to on a DC unless something went wrong. Change something on your main and it'd replicate, or hit a powershell command / use rsat and you're as good as on your remote dc.

I never understood people running so many DCs for such a small environment.

We don't have 100% reliable connectivity between sites. A few times a year we lose connection for a few hours to half a day sometimes to an off-site. Sometimes it's scheduled, sometimes it's not, and since the DC's are local, all internal services and clients keep working as if nothing happened till the link goes back up. People are screaming at network, not services.

Multiple DC's are about HA. It's actually simpler and more reliable to run more than less, they all have the same configuration.

Our environments are different: You have 15K @ 70 sites and a single domain. And it sounds like your services are centralized around maybe 3 datacenters? Many of your sites don't run services locally, and do require an external network to function.

We have 2 domains of about 3k users each on 4 sites, and run services locally. With two DC's you always have DNS & Authentication at each site, for each domain. Our sites don't require an external network to function.

If we were larger with more sites that ran services, we might go down to 1 per site, with an off-site backup for DNS, but with a datacenter license it's free and lower latency/local net traversal is always better. If you can run 2 DC's per site then why not is a better question. It's not like they're resource hogs.

That is why we run so many DC's, and unless something is really screwed up, it's no less secure or more difficult than running 1.

[-]

Physics_Prop@reddit

We don't allow privileged access like DA, rdp or ssh from a remote site. You must be on a privileged management network on a jump box that is tightly controlled.

My concern is physical, someone can walk in, boot off a usb, and they have the domain.

What connectivity issues do you have? We look at it as... no power/Internet... nobody is working anyways.

[-]

jrichey98@reddit

You are correct in you first assertion. But it's about services as well as users. All our apps would also go down if they lost Auth/DNS.

No power, no one is working, but we have a bldg generater. No internet, people can still access sharepoint / files / internal email / our vendor apps, etc... A lot can still happen.

We have engineers at all those sites, but physical seperation doesn't necessarily mean logical seperation. The right person can get to where they need if they need to work on a system.

Not saying that model is best for everyone. If we were manned for only a few central datacenters and had to support 70 sites, we'd probably have to tell them they're SOL without network.

As it is right now, management want's a network interruption to have as little impact as possible.

[-]

thortgot@reddit

Distance between sites and how much auth traffic you have are key factors in how many DCs you need.

RODCs don't add a significant amount of risk if you are protecting your hypervisors and VMs reasonably (FDE, monitoring, DRAC etc.)

Personally, shifting toward Entra Joined where possible is a much better alternative. PRT tokens are dramatically more secure than Kerberos auth.

[-]

Physics_Prop@reddit

Yes, we do 2x US East, 1x US West

RODCs were considered, but we weren't really noticing any delays in auth. Maintaining a hardware stack would be kinda silly. Kerberos is not as chatty as something like ldap where you are throwing passwords around.

Current org is cloud only, SAML/OAuth/PRTs are better in every way. We still technically have DCs for some legacy apps, but no line of sight from workstations.

[-]

Sajem@reddit

I never understood people running so many DCs for such a small environment

I think it probably comes down to absolute crap WAN connections.

We aren't a huge company, but we do have about 150 sites, we have two DC's in our Data Center and two in Azure and our SD-WAN is over fast internet.

[-]

jrichey98@reddit

Running DCs at each site would be an unacceptable level of risk.

No. It's the same profile for your endpoint protection, and since they replicate with each other. If you can compromise any then you can compromise one, so there's no difference in threat.

... we couldn't control each site like we do our datacenters.

I never understood people running so many DCs for such a small environment.

Multiple DC's are about HA. It's actually simpler and more reliable to run more than less, they all have the same configuration.

That is why we run so many DC's, and unless something is really screwed up, it's no less secure or more difficult than running 1.

[-]

RichardJimmy48@reddit

With 5 domains, you should be surprised that it's only 18 domain controllers.

[-]

anonpf@reddit

What do you mean migrate?

[-]

HarlanGames@reddit (OP)

Demote 2012r2 server promote and migrate fsmo roles to a 2016 server

[-]

HarlanGames@reddit (OP)

We didn’t get licensing for it unfortunately, it’s in planning

[-]

Physics_Prop@reddit

Are you a nonprofit? The license cost should be trivial compared to the cost of upgrading again.

[-]

HarlanGames@reddit (OP)

No, it just wasn’t budgeted this year or planned prior

[-]

Physics_Prop@reddit

What, you can't cough up $1000 to not deploy an end of support server in production?

How did you even get this licence?

[-]

HarlanGames@reddit (OP)

Our cluster would have costed a lot more than $1k.

[-]