Need Help: Admin Deleted our Primary DNS Zone when they meant to Refresh it
Posted by Krazie8s@reddit | sysadmin | View on Reddit | 60 comments
Our Primary DNS Zone was deleted. We have the Recycle bin enabled and I didn't see the Zone inside the immediate bin. After doing some digging with powershell i located the zone and its no longer found in the deleted items. The zone now shows with the list of remaining zones only has ...delted-my-zone-.org I suspect the zone is neither dead nor re-animated now so I'm thinking the next option is to use Veeam to recover it however there seems to be different approaches to this.
Option 1: Mount a recent backup offline(not on the network) and login in DSRM and then export the zone. Login to one of the domain controllers and re-import (Assuming it doesnt conflict with the deleted one in its current state...) And deal with any fall out of missing objects.
Option 2: Attempt to recreate the Zone then use Veeam to restore individual objects into the zone (Again assuming it can do this and not conflict with the "Zombie" deleted zone).
Option 3: Full Authoritative Restore of one of the domain controllers and force Replication then deal with the fall out of any new objects created since the backup.
Am I missing anyting? Is there a special process to delete the now "Zombie Zone" before attempting restoration?
RevLoveJoy@reddit
Preach. How you handle the crisis is often 95% of the solution.
My very first thought after your first attempt seemed to work yet no zone. DNS loves a restart to clear cache and re-read it's object inputs.
Adam_Kearn@reddit
Are you able to spin up a DC from a backup and export the DNS ZONE?
Sounds like a nightmare…..what happened to read only Friday
subsvenhurt@reddit
the. Deleted- prefix is your problem right now, DNS service won't load it because the object name is still in its tombstoned format even after Restore-ADObject runs. what worked for me in a similar situation was following up with Rename-ADObject to strip that prefix and get, it back to the actual zone name, then restart the DNS Server service on the DC holding the zone.
ThomasTrain87@reddit
I’ve recovered from this cluster before in a multinational company with over 100 DCs across 35 sites.. it’s ugly but move fast!
If you have any one of the DCs that still has the zone active:
1), immediately convert it to a primary zone! Salvage the zone data.
2) point all your other DCs DNS to that DC with the surviving, now primary zone and get replication healthy again. You will have to allow the ad integrated zone deletion to properly replicate through to all DCs.
3)once replication is healthy again and the zone deletion has successfully replicated, then convert the primary zone back to an ad integrated zone and allow it to replicate to all the DCs
4). Once replicated, restore the DNS resolver configuration on all the DCs.
Alternatively, attempt an authoritative restore but I believe you will still have the replication problems as with your primary zone deleted, DNS is now non-existent and none of the DCs know how to contact their partners anymore.
CrazyEntertainment86@reddit
This is terrific advice and hope the OP can recover, a good reason to have other secondary tools like “shudder” qwest recovery for Active Directory….
But on your last point wouldn’t all the DC’s in the domain be able to resolve each other ftnrough primary DNS fallback since it’s the primary zone of the “Forrest” or maybe just domain which would make a difference.
ThomasTrain87@reddit
It depends on how the admin set it up and how hierarchical their layout design is.
In most deployments I’ve seen, admins typically configure DNS on each DC to point to itself and then one other DC, typically in the same site to minimize wan network traffic. Ordinarily this works great, however, what happens is once the AD integrated domain is deleted, it replicates rapidly to all DCs in that site. Then it will wait for the next replication cycle for out of site replication (usually 15 minutes) and then replicate the deletion there, and in and on. Now because the DC DNS lookups are usually cached for around 30-60 minutes for the TTL of the record, this allows the deletion to sync throughout the AD environment, but then the moment the TTL expires from cache on each DC, the DCs can no longer perform a lookup for the DNS zone because it’s gone and no longer know the IP address for their DC peers and boom, your replication now falls over like dominoes, one DC or site at a time.
That is precisely why step 1 is important: ‘hopefully’ preserve the zone data on one DC. Then Step 2 effectively restores the ability of the DCs to make DNS queries against the AD domain name and be able resolve the IP and SRV records of the DCs for replication to be able to resume. Step 4 simply puts everything back the way it was before the bumper fire began.
CrazyEntertainment86@reddit
Certainly not a great situation, seems like AD should have a fallback zone for replication in such a scenario so you couldn’t “delete” the primary dns zone but would have to swap it instead over a series of actions. Seems like a punitive oops click in this scenario.
ThomasTrain87@reddit
Most of the newer versions since 2016 have a lot of warnings but don’t actually prevent you from deleting the primary zone. IIRC, Microsoft introduced guidance to implement delete protection on your primary zone but you have to manually implement it to enable it. It is usually one of the recommendations and findings on their best practices reports.
Snot-p@reddit
As an idiot who has deleted the primary zone and recovered exactly as the main comment above stated - 2016 really didn't guardrail me from making the dumbass mistake when I was younger and dumber haha.
You should've felt the relief that came over me when I thought wait why don't I just go to another DC and get the DNS I just deleted off there lmao
DickStripper@reddit
My PowerShell guy deleted 200+ prod service accounts one fine day. I opened MikeDellQuest AA restored them within 90 seconds.
Exceptional tool. Shitty company.
DailonMarkMann@reddit
Good stuff!
PawnF4@reddit
I would honestly just restore from backup if that’s feasible. Don’t know how large or complex your environment is so that may not be simple, but hopefully this DC is only a DC.
Cooleb09@reddit
Restoring DCs is an easy way to fuck up your domain.
disclosure5@reddit
This is a Reddit meme at this point.
There's been data lost. That data is in backups. Restoring that data is an entirely normal thing to do.
tastyratz@reddit
Of course it can be done...
If you want to spend all day rotating machine keys because a bunch of random endpoints broke trust.
Presumably next day shouldn't be TOO bad it's more that the users will find you, you won't find the users. a full restore of a DC into prod may also just... come back up and sync the zones.
Might just be able to restore %windir%\System32\Dns without sledgehammering a whole domain.
Cooleb09@reddit
Blindly restoring a DC will cause issues, and make the headache worse My point wasn't that it couldn't be done, just that it's an easy way to foot-gun yourself.
There are good ways to recover the data from backups (restore the DC with no network access, export the zone data, re-import into the healthy DC), but someone who's posting here for actual advice may not realize that there's more to it then 'press the button in the Veeam console'.
Mojo_Rising@reddit
Just pondering, would that work if you restored all the other DCs at the same time?
Cormacolinde@reddit
Not with Veeam, if guest processing is correctly configured. Veeam will automatically restore the DC in non-authoritative mode and boot it in DSRM. There you can use ntdsutil to mark the dnsdomainzone object as authoritative and recover.
First_Slide3870@reddit
Yes, do not restore if you’re running other DC. If it’s the only DC then let er’ rip.
Mojo_Rising@reddit
Just pondering, would that work if you restored all the other DCs at the same time?
BrainWaveCC@reddit
You aren't backing up your DNS zones independently?
tepitokura@reddit
How can you automate that?
BrainWaveCC@reddit
There is the DNSCMD command from long ago resource kits, and there is Powershell.
Such_Field_3294@reddit
Worth noting for anyone reading this later, make sure you verify SOA and NS records after the restore too. Those can get weird after a zone recovery and cause subtle replication issues that dont surface immediately.
CircularSeasoning@reddit
Why do I get the feeling the admin's name is "Claude". Heh.
info_solutions@reddit
" Perfect ! I successfully deleted all your organisation ! Is there anything else i can do now ? "
entropic@reddit
You're right to push back...
blbd@reddit
You can't blame the cheeky French admin for having a flair for acts of sabotage!
CircularSeasoning@reddit
Haha. I mean, Claude is by an American company but I get the name is French origin. Mistral is the actual French AI.
Omg, what, look at this:
https://en.wikipedia.org/wiki/Claude_(given_name)
It all makes sense now.
epsiblivion@reddit
“You’re absolutely right!”
FilthyeeMcNasty@reddit
Right. So many of these comments are either obviously inexperienced or Ai driven decisions. One reason why i reason to hire only based on education and certifications. I make candidates “show me”, not TALK their way through it.
This is one reason why so many cybersecurity events are happening. Too many bullshitters, who know buzzwords and get play cyber then actual cyber.
tepitokura@reddit
Have you had any luck?
macro_franco_kai@reddit
Congratulation to the management who hired amateurs/imposters since they are cheap :)
SIGN_JULIO@reddit
Prepare 3 envelopes....
commiecat@reddit
Late, but some other tips about recovery and preventing it to begin with.
First, you can flag DNS Zones as "Protect from accidental deletion" via PowerShell. This isn't enabled by default and needs to be set. Get the zone's distinguished name and use
Set-ADObjectto set the parameterProtectedFromAccidentalDeletionto$true. Obviously you'll need to revert that with the same process to intentionally delete a zone with this set.Next, when trying to do AD recovery, the location will vary, and could be nested within
ForestDnsZones,DomainDnsZones, or theConfigurationcontainer.When the zone is deleted, its name is changed to prepend
..Deleted-to the beginning. This needs to be considered when searching, and the zone will be restored with that same name. Restore the zone with the deleted prefix, recover all the records, and then rename the zone when you're ready to make it 'live' again.Krixim@reddit
Don’t do an authoritative restore unless this is truly catastrophic. If the zone objects are still visible in PowerShell, I’d clean up the zombie state first and attempt a proper AD object restore again before rolling back a DC. Worst case, restoring just the DNS application partition from backup is a lot less painful than dealing with AD fallout from option 3.
Fallingdamage@reddit
Use powershell to check your zones? I know that any ZoneScopes or ResolutionPolicies never show up in the snap-in, you have to look for them via powershell. Maybe something is off with the snap-in and it wont display things that were restored?
Use Get-DNSServerZone or Get-DNSServerZoneScope and see if it shows up there.
Now that you've restored the deleted zone, are hosts resolving DNS queries properly even though you cant see the zone?
Cormacolinde@reddit
This is what I would do. This will only work if Veeam is properly configured with guest processing for backing up your domain controllers. You will want to do a partial authoritative restore of the Zone. Read the Veeam documentation carefully, the section is called “Restoring due to Active Directory Corruption”: https://www.veeam.com/kb2119
Open a command prompt and start NTDSUTIL
activate instance ntds authoritative restore restore subtree DC=DOMAIN.COM,CN=MicrosoftDNS,DC=DomainDNSZones,DC=DOMAIN,DC=COM
Then reboot the server
If you’re unfamiliar with this, I highly recommend you hire a specialized consultant to do this properly. You should also take some precautions, like taking new backups of your DCs in Veeam before proceeding. Make sure those are guest-aware and that netdom /showbackups confirms a backup has been taken.
xXFl1ppyXx@reddit
going forward from 2003, the dns is stored under
"DC=DomainDnsZones,DC=domain,DC=com"
and
"DC=ForestDnsZones,DC=comain,DC=com"
"DC=DOMAIN.COM,CN=MicrosoftDNS,DC=DomainDNSZones,DC=DOMAIN,DC=COM" probably will get you nowhere
Cormacolinde@reddit
Thanks for the correction I wrote this from memory and don’t have access to my lab.
ChangeWindowZombie@reddit
Good luck op, lots of solid advice here.
Once you are out of the woods on this, it would be good to start exporting all DNS zones to a secure location that is backed up daily.
To export a single zone: DnsCmd YourServerName /ZoneExport YourZoneName YourBackupFileName
Example: DnsCmd Houston /ZoneExport Wehaveaproblem.com Wehaveaproblem.com.backup
xXFl1ppyXx@reddit
you could grab the ad db from backup, mount it, with
connect via adsi to that, now running, database backup, open adsiedit and connect to that running database by selecting a different port (51389 in this case) to the custom name
"DC=DomainDnsZones,DC=domain,DC=com"
and
"DC=ForestDnsZones,DC=domain,DC=com"
where you'll find all your dns records.
you can export those records into a file with ldifde and use that file to import back into your domain
https://learn.microsoft.com/de-de/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/cc731033(v=ws.11)
to Export:
to import:
dawg4prez@reddit
I would see if you can use Veeam Active Directory Explorer to restore the zone to the DC. I have never done this, but it’s worth looking at. You need to enable “Advanced Features” to see the Integrated DNS node in AD Explorer.
good luck!
techb00mer@reddit
This type of thing weirdly reminds me of this story:
https://www.phishingforanswers.com/blog/how-ghana-saved-a-conglomerate-from-cyberattack
FKFnz@reddit
That's one of my favourite IT disaster stories to tell newbies.
UninvestedCuriosity@reddit
Maybe setup some scheduled zone exports in PowerShell for the future as insurance against this colleague.
pioneersohpioneers@reddit
As someone who just completed a huge over haul of my orgs internal DNS, I feel for you. I deleted or migrated so many zones and subzones out of AD DNS and frequently thought "shit did I just delete the wrong one?"
Godspeed Bud
russellvt@reddit
Should be able to xfer it from a secondary and restore it to the primary fairly easily.
It may not be as-clean, but at least you will have the data.
tch2349987@reddit
It’s always good to double check everything and test before executing something critical.
Cue_The_Duckboats@reddit
This comment made me irrationally angry
Conscious-Arm-6298@reddit
Just check the things before bro fr
KingSummo@reddit
Thanks Genius
Less-Philosophy-1978@reddit
how can someone make that mistake?? if they did you guys need real evaluation
Xibby@reddit
Ouch.
This is why I have a bunch of scripts backing up DNS. I’ve done that before in the days when you could run to the server room and yank cables before Active Directory replicates. 😂
Internal Windows DNS, Azure, Route 53, GoDaddy (FML), all getting a nightly backup of DNS zones onto a Bind zone file, easily imported into any provider.
Very handy when your customer decides to move their website to Wix or whatever and calls to tell you they aren’t getting email. 🤦🏻♂️
Bart_Yellowbeard@reddit
Too critical to fuck around with. Turn off ALL DC's, restore whichever held most of the roles from before the error, rebuild all the other DCs. Unless you have a large or complicated AD environment, but even then, multiple sites, unusual trusts, might still be easier to just rebuild than repair.
PhantomWang@reddit
Given OP's simple AD environment this is definitely the right call. Any data lost since the last backup is nothing compared to fucking up AD.
Crazy-Rest5026@reddit
Sounds like you need a drink after this day.
Option1 sounds promising. Or option 3. I think a veeam instant recovery might work if you got it backing up in veeam. But with it being a DC taking a risk.
DailonMarkMann@reddit
If you can get into Veeam, does it have a copy of the zone? I’ve restored other objects out of veeam, but not dns. That would be my first move. Second move: full restore of dc that has the PDC role. Good luck. Let us know.
newworldlife@reddit
DNS issues are always the ones that turn into psychological warfare after a while. Everything starts failing in ways that make no sense and you stop trusting half the environment.
tj818@reddit
Could do an instant recovery in veeam and try to get it from there. I know CommVault has dns zone recovery built in their AD agent. Don’t think veeam has that functionality.