Server 2016 reboots blew up domain. .

Posted by VNJCinPA@reddit | sysadmin | View on Reddit | 25 comments

I rebooted Sunday morning, and now the two domain controllers aren't able to authenticate to each other with 3210 NETLOGON errors (can't find DC XXXX, might be another computer with the same name or the computer account is failing). The domain is very old (20 years). I've been working through cleaning up demoted domain controllers from previous admins. I've also upgraded the domain and forest functionality R2. I have run dcdiag /e /c /d /v on each DC, and they passed domain checks and replication checks successfully. However, DNS fails for DNS BASC (Basic) for each other domain controller. They are unable to resolve the name of the other, and give (name unavailable) VALID in response. Each DC points to the other for DNS first and itself second. I also recently set up certificate auto enrollment prior to these reboots. I'm getting nothing from the logs. I've ran PortQryUI for Domains and Trusts, and port UDP 135 fails, 137 is filtered even though both DC's have Windows firewall off for now. I've also stopped KDC and reset the computer account on the non-FSMO system to make sure it wasn't the computer account, but the issue remains. I've had a paid Sev A ticket in to Microsoft for over 48 hours now but they haven't called back stating a very high number of tickets for Windows Server, so I figured I'd reach out here and see if anybody had any suggestions? My hunch is Kerberos, but things there still use NTLM for the most part. Symptoms are now that a SQL Reporting 15 server can't authenticate, and the two DCs can't validate each other despite being able to successfully replicate? Anybody have any recommendations? I'm at hour 30 since Sunday afternoon trying to solve this issue with no luck... Thanks in advance 😔

Reply to Post

25 Comments

[-]

_crowbarman_@reddit

I cut my teeth on DNS issues since Win2000. The basic method for fixing this in all situations: Point both DCs to one DNS server for now. Connect to that one dns server specifically in the DznS console. In the domain, look for the four _tcp, _gc, _ldap, and _mcdcs folders in your root domain and delete them fully. Look for the "A" host records in the root domain and delete them fully. Now run ipconfig /registerdns on both domain controllers, and restart the Netlogon service on both servers. You should see two A records appear for the host names of the servers and the four folders should be back. If you drill into the four folders you should see dns records in pairs for both servers. Force replication again on each box with repadmin /syncall /A /e /P Once all is working fine again change dns back the way you had it. If dns isn't registering on the re-registration step then I have also temporarily changed the "Secure" zone to allow unsecured updates until the records are populated as a troubleshooting step.

[-]

VNJCinPA@reddit (OP)

You were super helpful, I wish you the best! Thank you!!

[-]

VNJCinPA@reddit (OP)

Did what you'd said, it DID work but didn't solve the issue. What it did do though was give me some corrections that pointed to the underlying issue: One of the DC's lost all it's certificates (I think when an on-site employee issued a new self-cert...) Not completely fixed yet, but much closer, so thank you!

[-]

_crowbarman_@reddit

Great. Eliminating dns is always good because it's the cause of so many issues. Glad you on the right track.

[-]

VNJCinPA@reddit (OP)

Q: Looks like the _mcdcs folder is NOT delegated (no grey icon) and is just a plain folder under the main domain. Should I delete that folder and delegate it, then proceed?

[-]

_crowbarman_@reddit

If you are dealing with a single domain /forest situation (which I am guessing) then delegation isn't even needed. You can delete it and it should recreate undelegated. Alternatively, if you leave the delegated folder, then make sure the other zone's dns records are cleared out so they can be repopulated, and that the delegated folder has a reference to that domain controllers IP address (if that makes sense). But if it were me, I would probably delete the delegated folder and seperate mcdcs zone and then proceed with the-registration if in a single domain / forest scenario. Your choice. You could leave it to start.

[-]

VNJCinPA@reddit (OP)

It is single/single and all FSMO roles on one of them so this should all work. Thank you again!

[-]

_crowbarman_@reddit

Let me know how it turns out.

[-]

VNJCinPA@reddit (OP)

You've been awesome, and I'm working on this but have a new issue: No matter what I set DNS to, it still uses and registers ::1 in DNS, so the second DC points to itself. I do have the DisabledComponents 0xff on both servers it's still wanting to use itself.... If it I set it to obtain for IPv6, it still chooses ::1. Any thoughts here?

[-]

_crowbarman_@reddit

Did you go to the properties in the ipv6 protocol on the network adapter, and in the dns section make sure that DNS is set to blank? Don't choose automatic for the dns server address just set it to "Use this" and leave it empty.

[-]

VNJCinPA@reddit (OP)

I did, but it's still registering the adapter's DNS in Active Directory as ::1 even though I'm not listening on IPv6, Disabled Components and set to auto for v6.

[-]

_crowbarman_@reddit

If you go into tcp/ip settings for the ipv6 protocol. Go to advanced, dns tab, is the Register checkbox selected? If so ubcheck. Also you don't have any other network interfaces on the server right?

[-]

VNJCinPA@reddit (OP)

It is, but I thought I needed that to reg the IPv4 addressing? If you tick it in IPv4, it's on in IPv6, so it's all or none, and I thought I needed to do that as part of the DNS process above?

[-]

_crowbarman_@reddit

That could be right, it is per interface not protocol. I am pretty sure you must have DNS hard coded to ::1 on the ipv6 protocol of that server. Did you check? Similar to this. https://www.experts-exchange.com/questions/26845195/Can-I-remove-the-1-IPv6-loopback-address-as-the-Preferred-DNS-IP-on-the-IPv6-interface-of-my-Domain-Controllers.html

[-]

_crowbarman_@reddit

You can see here it doesn't matter whether delegation is there or not. You go need to delete the delegated zone as well as the subfolder to get rid of it. https://community.spiceworks.com/topic/2310568-_msdcs-domain-com-at-the-root-of-forward-lookup-zone

[-]

VNJCinPA@reddit (OP)

This is extremely actionable, it's what I was actually pursuing now so I'll do this. I have both DC's pointing to each other for DNS at the moment but this seems very very sound! I'll update you in a bit!

[-]

VNJCinPA@reddit (OP)

After 3 weeks and some great help here but not Microsoft Support, I've found the following: I have older Domains I'm updating, and they had Windows Firewall off. Recent updates change settings in the Firewall where even if it's disabled for all your network profiles, it blocks RPC traffic... The solve is to disable Windows Firewall service, then go into Windows Firewall and click Use Recommended Settings. It will error out because the service is disabled, but it also does some registry kung fu and stops blocking certain things or shouldn't be blocking anyhow when the Firewall is turned off. 3 weeks to find I just needed to click a button that seemingly doesn't apply to anything when it's disabled. Hope this helps others!

[-]

VNJCinPA@reddit (OP)

Indicators are NTLM authentication now. Working through auditing them now to track it down.

[-]

sammnz@reddit

Assuming you only have two domain controllers, I’d turn a DC off, seize roles for the other and reset the krbtgt account password https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/forest-recovery-guide/ad-forest-recovery-reset-the-krbtgt-password

[-]

VNJCinPA@reddit (OP)

I actually already did the first of this, now I'm getting ready to do the second to get rid of RC4 from that account. I am no longer getting those errors for krbtgt strong authentication at least, and I'm using that Microsoft script for the resets. It worked very well the first time it seems as the error went away.

[-]

VNJCinPA@reddit (OP)

Another note: I feel it may be IPv6 related... I made some progress. I used DisabledComponents 0xff for IPv6 and disabled Large Offload on the network adapters and they both began communicating errors at least. Before I didn't even get that. I might be on a good path here.. If anyone has any thoughts on how to NOT disable IPv6, I'd appreciate it. Or any other suggestions here...

[-]

VNJCinPA@reddit (OP)

I can also add that DCDIAG gives a few Access Denied (5) errors for KCCEvent, FRSEvent, SystemLog, and OutboundSecureChannels. Additionally, on OutboundSecureChannels I get could not query trusted domain, error 0x2, The system cannot find the file specified.

[-]

disclosure5@reddit

> Anybody have any recommendations? Here's what I'd do: Grab whatever you want to be the primary server. Perform a restore from backup to an isolated network to test this out before making it "live". Perform an "authoritative restore" so it's in charge of whatever isolated network it sees, then metadata cleanup every other Domain Controller. If it's not the primary role holder, seize the roles. You should be left with one happy device. If it works well, put it in production.

[-]

VNJCinPA@reddit (OP)

Great advice, and about where I'm at except for deciding which one to choose, and I do have the resources to do this. I'm going to start that process now but very much hoping my "2 hour response time" doesn't go much past 56 hours.... Or I stumble on the issue. Thank you!

[-]

disclosure5@reddit

> but very much hoping my "2 hour response time" doesn't go much past 56 hours. For a Sev A Microsoft ticket? They'll email you three days from now asking what OS you're running, then two minutes later send you an email chastising you for keeping them waiting and threaten to close the case.