I'm going through the account lockout from Hell
Posted by BoomSchtik@reddit | sysadmin | View on Reddit | 236 comments
I've been doing IT in one form or another for 30 years. I've never had a lockout problem like this. This is happening to my admin account, and it gets locked out just about constantly all day. I know the server that the locking out is happening on because of the lockout events on the DC.
- Server 2022 Datacenter running on VMWare
- This server runs our Azure AD sync
- This server is our PDQ Deploy and Inventory machine (Those services are stopped)
- Double and triple checked that there is NOT a service or scheduled task using my creds
- This has been going on for two weeks now
- It seems like a service, but I can NOT figure out which one.
- With PowerShell I wrote a script to find all .ini, .cfg and .xml files on my c: and search those for my username. It found two xml files that were task manager exports. The username was just a refernce to
and , not using my creds. - I've cleared credential manager and Windows Vault
- There are no mapped network drives,
- Backups are hypervisor based so there's nothing running in the guest OS in that regard
- I've tried the Netwrix Account Lockout Examiner and it didn't find anything useful.
- I've search all running services and asked Perplexity which ones might be using user impersonation. It gave me a list. I stopped the ones that it would let me stop, but that didn't have any affect.
As you can tell, I'm getting a bit desperate. I could really use a Reddit hive mind miracle.
Thanks!
black_tbs@reddit
I had a case once with a PC that I was troubleshooting, the admin login was left in memory and a stupid user clicked on the admin login without paying attention and typed his password until the other account was blocked...
Unseen_Cereal@reddit
Entra joined hybrid AD environment (and we only had our admin account), and I had a similar issue. My manager was a pathetic excuse of a leader, and did not help whatsoever. Ended up just creating a new account for me, didn't even care about searching for a root cause.
I was logged on to a laptop to help a user and I did a switch user a couple weeks prior, and forgot to fully log out. That's the only thing I could think of that could have caused this.
Dizzy_Bridge_794@reddit
Netwrix makes a password lockout analyzer that’s free
BoomSchtik@reddit (OP)
Check bullet point #10. :)
iiiiijoeyiiiii@reddit
I've seen Netwrix itself cause lockouts exactly like you're describing.
Recent_Carpenter8644@reddit
Really? How?
iiiiijoeyiiiii@reddit
I would assume it was auto launching after he restarted and it had his credentials stored from before he changed his password. I think I noticed they were coming from Netwrix when looking through his PCs event viewer. I think it even started up and was locking him out before he even got signed in sometimes. I never actually used Netwrix myself.
Recent_Carpenter8644@reddit
That's almost funny. How many bad passwords do you allow before a lockout there? I would have thought that app wouldn't try too many times, but I guess it could be authenticating to many DCs in turn to collect the information.
Dizzy_Bridge_794@reddit
Didn’t see that
gangaskan@reddit
Debug your net login.
I had to do this recently as well. Found out some turd was trying to brute force and overflow a server of ours.
pleaseguysomg@reddit
Get a manage engine AD Audit free trial. Monitor all your DC’s and use the lockout analyzer report
buy_chocolate_bars@reddit
Try the two steps below and report back.
1) Find out which IP/Servers the locks are coming from (i assume you know how)
2) Remove your windows user profile from those computers.
3
gr8bhere@reddit
Old WiFi password saved using network authentication
00Nygma@reddit
Had this happen with one of my accounts. I had manually entered my account on a casting device, screenbeam in this case, to connect to our Wi-Fi to test out the device. Unplugged it and didn't get back to it for sometime it got reconnected and boom lock outs started happening. I've also seen account lock outs happen with scan snap scanners setup over WiFi in our network.
BoomSchtik@reddit (OP)
We do use domain creds via RADIUS, but I've had lockouts when my computer was logged out and shut down, so I don't think that part of the issue. I wouldn't use my creds for wifi on any computer other than my own.
gr8bhere@reddit
WiFi on your phone? I’m only asking because it’s always been the cause of my invisible lockouts. Have to look at the radius logs to find it not the DC logs, very annoying.
Recent_Carpenter8644@reddit
It's usually phone wifi that causes it for us, but it would be uncommon to use admin credentials for wifi, wouldn't it?
gr8bhere@reddit
Yeah but the symptoms are so familiar I’d double check for sanity sake
Comptonistic@reddit
Do you have a smart watch? Could your phone be sharing old wifi credentials with it?
BiteMaJobby@reddit
Any time I have seen this issue its always been the users mobile phone trying the previous password on Wifi
BoomSchtik@reddit (OP)
I don’t use my admin account for WiFi, so it’s not that in this case.
Recent_Carpenter8644@reddit
Have you checked for event 4625's from before the password change if your event logs go back that far? You don't get a lockout from old passwords till the password is changed a couple more times. If you could prove that you were already getting 4625's before that, it would show that the problem is not recent, possibly years old, which might be a clue.
Outrageous-Chip-1319@reddit
Cell phone with old password to network?
BoomSchtik@reddit (OP)
No, this is locking out when I'm at home and sleeping, but I would only use my regular account for wifi, not my admin account.
Recent_Carpenter8644@reddit
Is there any possibility that it's someone else's phone? I've seen that happen. And lockouts don't happen with bad passwords until, I think, two password changes later, so whatever it is could have had that password entered a very long time ago.
iiiiijoeyiiiii@reddit
Did you look in to the Netwrix thing I mentioned? It just sounds so familiar to me and the issue turned out to be Netwrix causing the lockouts. Were you using Netwrix for something else before your lockouts started and it was right after you changed your admin password?
itiscodeman@reddit
Seriously , why not make a new user ?
BoomSchtik@reddit (OP)
I'm sure that the fix is right around the next bend or over the next dune. :P
Future_Zone@reddit
Rename the account. A ahem friend of mine ran into the same issue at one point and renamed the account. Problem solved.
Recent_Carpenter8644@reddit
Symptom of problem solved. Something would still be failing.
RandyCoreyLahey@reddit
it doesnt have a public ip does it? i once tracked down random lockouts to a fucked up firewall policy by the isp after a router switchover. rdp open to internet so hammered some accounts
Infinite_Somewhere58@reddit
There is an AD lockout tool from Microsoft you can install on the DC and it tells you exactly what computer is locking out your account:
https://www.microsoft.com/en-us/download/details.aspx?id=18465
Sufficient_Bet_3624@reddit
Do you use a radius server to authenticate to WiFi? If so have you forgot and added back the network? Have you looked at the sign in audit log?
BD98TJ@reddit
Download the account lockout status tool from Microsoft. Unlock your account. Run the tool against your account every hour or so until your account is locked. The tool will show which DC the account got locked out on and the exact time of the lockout. Connect to that DC and find the lockout entry in the security event viewer log by using the account lockout time to find the event. Typically the log will show the ip address from the computer the failed attempts came from. Connect to that computer and figure out what’s going on. If you are logged into a bunch of servers you didn’t log out of with the old password this will happen also.
No_Stretch312@reddit
I know this is a very stupid question especially for this sub lol but have you restarted the server?
UseMoreHops@reddit
LOL
BoomSchtik@reddit (OP)
Oh yes... many reboots. :)
kuahara@reddit
A server performing the account lockout does not mean that the event causing the lockout happened on that server. It could be a login ok another computer that was done using the previous password.
Most of the time that I see this, it is usually from someone that closed an RDP session without actually logging off of the computer and then went and changed their password sometime later.
That computer winds up submitting bad creds back to the DC until the account is locked out. You can unlock it, but that machine will just keep locking it out again.
I am in bed right now, but I have a Powershell script that will expose which computer on my network is doing this whenever users have this problem.
jstarr20052005@reddit
I would love a solution of a PowerShell script. This happens all the time and is always such a hassle to track down what is causing the lockouts. Would you share?
kuahara@reddit
So there's a few ways to tackle this.
}
That will display recent authentications to the DC. You can use a script that loops through all computers in AD and uses the quser command to see where they are actively logged in, but if you have a large environment, this will take a really long time to run. I have around 2300 machines, so I would never do this. However, if you have some idea of where it might be, you can target an OU with that. Example, you know it's one of the application servers and you only have a handful of those, you can quser each AD computer in your app server OU.
A more helpful approach if it is an issue you're dealing with right now is to look at recent lockout events. If a user is telling me that their account keeps getting locked out and I just unlocked the thing 10 minutes ago, then I do this instead, which is super helpful:
This shows me every account that was locked out in the last 20 minutes (adjust time for whatever you need). It will show you the DC that performed the lockout and which computer triggered it.
BoomSchtik@reddit (OP)
I appreciate you sharing your scripts. Part of the problem is that the CallerComputer which your script returns is ALWAYS the same computer/server. Everything seems to point to this one server, but I still can't figure out what's causing it. These are from the last 4 hours.
10/17/2025 2:52:05 PM adm.account ProblemChildServer DC01
10/17/2025 2:52:05 PM adm.account ProblemChildServer DC02
10/17/2025 2:23:24 PM adm.account ProblemChildServer DC02
10/17/2025 2:23:24 PM adm.account ProblemChildServer DC01
10/17/2025 2:05:02 PM adm.account ProblemChildServer DC01
10/17/2025 2:05:02 PM adm.account ProblemChildServer DC02
10/17/2025 1:31:10 PM adm.account ProblemChildServer DC02
kuahara@reddit
Can you run this and share the output?
nltest /server:ProblemChildServer /sc_query:domain
Get-ScheduledTask | Where-Object { $_.Principal.UserId -match "adm.account" } | Select-Object TaskName,TaskPath,State,LastRunTime
Get-WmiObject Win32_Service | Where-Object { $_.StartName -match "adm.account" } | Select-Object Name,DisplayName,StartName,State
wevtutil qe Security /q:"[System[(EventID=4625)]] and [EventData[Data[@Name='TargetUserName']='adm.account']]" /f:text /c:10
GreyAzazel@reddit
Had this exact thing happen to me. Thanks for sharing so I don't need to!
noodlyman@reddit
Maybe it's everything else that you need to reboot
Comprehensive_Lab959@reddit
This ^. Once I spent hours troubleshooting a strange lockout issue like this. Kicked myself for not doing the most basic troubleshooting step. After a reboot, never had the issue again.
1stUserEver@reddit
Dealt with this weeks ago on 2019 server. admin account locks seconds after unlock. we just disabled that account. check for services running under the account. nothing worked for us so we disabled it.
killasrspike@reddit
This
Ihaveasmallwang@reddit
It’s not necessarily a service. It could be something as simple (not always simple) as credentials cached somewhere.
When is the last time you changed your password? Does it line up at all with when this started?
It could be a stale Kerberos ticket as well. Have you tried purging those on this server as well as everywhere else?
Or it could even be another machine trying to connect to this particular server with old credentials.
When is the last time this server was rebooted?
pakman82@reddit
Great points. * Also like to suggest azure / online. If the server passes stuff back and forth to azure, it'll be "a middle man" in the pain process.
BoomSchtik@reddit (OP)
The lockouts definitely correspond with my last password change. It's been about 2 weeks.
I haven't done anything with Kerberos. I'll search that up in Perplexity.
I don't think it's coming from another machine. The logs on the DC seem to make it clear that it's happening from this one server.
I've rebooted it at least a dozen times in the last 2 weeks.
Ihaveasmallwang@reddit
You don’t need to search it up in perplexity. Just log into the server, or anything else you’ve logged into, open a command prompt and type “klist purge”.
It’s a safe command to run.
The logs on the DC can absolutely make it appear like it’s happening on machine A even when it’s machine B connecting to machine A. That’s why account lockouts are notoriously hard to track down. This can happen if you’re trying to RDP into machine A from machine B, or run powershell commands on machine A from machine B, or try to access something like the C$ share from another machine, or anything similar.
BoomSchtik@reddit (OP)
klist purge didn't fix it unfortunately.
kuahara@reddit
OP, pop open powershell on a DC (or somewhere you have RSAT and AD module loaded) and run this:
$results | Sort-Object TimeCreated -Descending | Format-Table -AutoSize
BoomSchtik@reddit (OP)
This doesn't show any more than the other tools. I know that the account is getting locked and I know that the CallerComputer is always the same server I mentioned in the OP.
Ihaveasmallwang@reddit
Did you only run it on that one server? Or everywhere you’ve logged in with that username?
blueeggsandketchup@reddit
This maybe something here.
You mentioned PDQ runs on the machine. Is a software share being accessed by your admin account from another machine? You may also want to turn on login failure auditing on this server.
BoomSchtik@reddit (OP)
I appreciate your insights. Does klist purge need to run in the context of the user, or can I use PDQ to run that command on all of our servers as SYSTEM?
Ihaveasmallwang@reddit
Run it in the context of your user on all the machines you log into or would touch that machine from.
Helpful_Friend_@reddit
This reminds me a lot of an issue I had at a client, where I had logged into their print server and not logged out, just exited rdp. 1-2 weeks after I started being instantly locked out at that client on any other server I tried to log in on, because I had changed my password and the server was stuck re-authing against AD with my old password, leading to contious lock outs.
Might want to see if your account is still logged on but disconneded on somewhere, can just run an invoke-command against your servers, if winrm is open
NextSouceIT@reddit
Every time you log in or access a network share, you use Kerberos.
BoomSchtik@reddit (OP)
I just meant troubleshooting wise.
Morkai@reddit
I think OP meant they haven't carried out any troubleshooting specific to Kerberos. Not that they've never used Kerberos.
Geminii27@reddit
Are the logs giving timestamps for the login attempts? Could a script monitor the relevant logs, and when it sees a login attempt for your account, take a snapshot of everything that's running at that moment?
Compare snapshots, and it might narrow down which processes are the most likely culprits.
bindermichi@reddit
Did you use that account to configure a service somewhere?
The logs should tell you the source of the login attempts
armada127@reddit
Did you leave yourself logged into a VM… it’s always a VM
darklordpotty@reddit
Its a stale kerberos ticket 100%
badaz06@reddit
This. Sign out of every device you can think of. Phone, iPad, desktop, Mac, RDP's, everything.
mister-pikkles@reddit
2 things immediately come to mind. 1) Is the Azure AD sync using your credentials? (Hopefully you're using a service account, but I'd double check.) 2) Have you checked Azure AD logs (assuming you're hybrid) to ensure someone isn't trying to brute force your account externally?
BoomSchtik@reddit (OP)
Both have been checked and rechecked.
Thanks for your efforts though.
CaptainZhon@reddit
Do you know what system is locking out your account? Is it possible that you changed your password and something like Solarwinds or other software has your account information cached? You can also go into aduc and limit what systems your account can log into as a means to help find the system that is locking out your account. I will tell you my account lockout story from hell and maybe that will help-
My director called me saying her account kept locking out. It was impacting her ability to work because she couldn’t check email or login to teams/office 365 while it was locked. I ran the Mslockout tool and a free ware tool to see what machine was locking her account- it was our AD Sync server - that was weird. After a week I just gave her another account to use because I couldn’t easily find it. I was a manager at the time and while my AD skills were good I also enlisted our AD SME to look at it and confident that they would find it- nope. I ran event comb on the sync server and DCs and nothing but the lockout call from the AD sync server. Fucking weird.
A month goes by- now it’s on the back burner as my director is working and tasks are pilling up. I was troubleshooting an issue in our monitoring software Solarwinds and found an office365 task that used her username and password. Yes the account started to lockout after she changed her pw, yes she said she changed it everywhere (she an ex exchange admin she knows her stuff) but she forgot about this task in Solarwinds and it was “test”- and it was locking out her account. Disabled the task, unlocked her account and no more lockouts.
Mg7Si8O22OH2@reddit
What information are you seeing in the Lockout Events in Event Viewer?
If you have access to the DC you can enable the netlogon debug log with this: Nltest /DBFlag:2080FFFF
That will generate a log file at %windir%\debug\netlogon.log with additional information you can use to narrow down the source of the lockouts. After enabling the flag you will need to wait for another lockout and then check the log file.
To turn it back off run: nltest /dbflag:0x0 or delete the key it created here: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\DBFlag
BoomSchtik@reddit (OP)
I have enabled netlogon logging on one of the DC's that the account lockout tool shows bad logins. Either there's nothing there, or I'm not looking for the right thing when trying to parse it out.
Mg7Si8O22OH2@reddit
If the log doesn't show any authentication attempts and the account is still getting locked out then it is likely a different server processing the authentication request.
If the authentication attempts are occurring on a different server then you may need to turn on logging on all other potential servers.
If you are using Lockoutstatus.exe it may not help with narrowing down the source of the lockouts. Depending on how frequently your DCs replicate its not likely you can check and happen to catch it locked out on one specific DC.
xMcRaemanx@reddit
Are you sure the lockout is coming from that server or is it just showing that server because it's the ad sync source?
Have you checked failed logins in Azure since the password change?
coffee_ice@reddit
Several comments have suggested cell phone or some other device and you keep responding that "It's not related to wireless."
That's a little bit baffling, because none of those comments (including mine) are suggesting it's a wireless issue. Why would you think that? Cell phones use mobile networks. They won't be on wifi unless you join them to wifi.
If you are seeing the lockout coming from your wireless controller, it means likely "something" is passing the failed credentials through the controller.
Joining wifi is not the same thing as passing credentials through to another server. That is not the same thing as failing to join the wlan or sending a bad password to the controller.
Again, cell phones are not on your wireless by default, and they won't join wireless unless you configure that, and an account lockout doesn't necessarily have anything to do with joining wireless.
This isn't about joining wireless and has nothing to do with your wifi authentication (unless somewhere on wifi you have configured something to join with your old admin password, but I don't think anyone suggested that) . It's about some app or service that happens to be coming through a wireless device.
Since it happens overnight here is what I think is happening. When you have processes running during the day and you are using your admin account daily, the lockout isn't always hitting because it's getting reset by your successful logins.
When it happens overnight, and you aren't logging in or running those processes, that means likely some idle device is in the office passing the failing credentials. It's locking out because it's eventually hitting the lockout threshold because you are not on, logging in and thereby resetting the lockout count.
That device is clearly joined to wifi. Again, joining wifi is not the same thing as passing other system credentials to another server through the wifi connection.
You need to think back through every step of every troubleshooting or admin thing you have done since your last admin password reset. Some device out there has your old password and it's failing to login to email or some app.
You ran your admin account for testing some app, possibly for one of your users and you immediately forgot about it once the issue was fixed.
Find the rogue device and you have your answer.
Seikai83@reddit
If you don’t need anything running automatically with your account, why not just change your account name and see if anything breaks?
kloeckwerx@reddit
Do you happen to have any open browsers in an rdp session on this server?
slav3269@reddit
Have you tried not having your account locked out by applying FGPP?
BoomSchtik@reddit (OP)
Are you suggesting moving my admin account to an OU and applying a password policy GPO to it that is different than the rest of the domain?
slav3269@reddit
Yes, just to stop the lockouts.
I also generally recommend disabling account lockout, but there are few precautions.
The__Relentless@reddit
Is your cell phone using any credentials that may have been changed? Email or Wifi?
BoomSchtik@reddit (OP)
This isn't related to wireless. When accounts get locked out on the wireless, we see the wireless controller (Aruba) as the lockout source, not the server in question.
wetnap00@reddit
Do you log into our somehow use the locked account with PDQ? I’ve seen PDQ consistently lock accounts for no reason and support was baffled.
BoomSchtik@reddit (OP)
That was my first suspicion. I've removed my creds, stopped and disabled the services and it's still happening.
tepitokura@reddit
And old cellphone with the email client using the old credentials.
BoomSchtik@reddit (OP)
This isn't related to wireless. When accounts get locked out on the wireless, we see the wireless controller (Aruba) as the lockout source, not the server in question.
somniforousalmondeye@reddit
Is it happening on a schedule? I just fixed a similar issue and it was occurring every 2 hours in the dot. Ended up being our E backup service that had cached old credentials from a restore job.
BoomSchtik@reddit (OP)
These are the lockouts from the last four hours:
10/17/2025 2:52:05 PM adm.account ProblemChildServer DC01
10/17/2025 2:52:05 PM adm.account ProblemChildServer DC02
10/17/2025 2:23:24 PM adm.account ProblemChildServer DC02
10/17/2025 2:23:24 PM adm.account ProblemChildServer DC01
10/17/2025 2:05:02 PM adm.account ProblemChildServer DC01
10/17/2025 2:05:02 PM adm.account ProblemChildServer DC02
10/17/2025 1:31:10 PM adm.account ProblemChildServer DC02
coffee_ice@reddit
have you ever checked work email from your phone, home pc or ipad?
BoomSchtik@reddit (OP)
This isn't related to wireless. When accounts get locked out on the wireless, we see the wireless controller (Aruba) as the lockout source, not the server in question.
JankyJawn@reddit
Bro i had this same shit like a year ago. I wish I could recall what it was. Some sort of sync issue i think.
BoomSchtik@reddit (OP)
I created an Enterprise Admin service account for Microsoft Entra Connect Sync to use to try to eliminate the possibility of it coming from that.
etherez@reddit
Could it be a scheduled task?
BoomSchtik@reddit (OP)
I don't think so. I can't see anything that makes me think that.
Few_World6254@reddit
A) Why is your account deeply rooted in the administration of the domain? Is it an admin account? It should be an easy account to blow it away and spin up another one. Is this also the primary account you log in and do work with? I tend to agree with others, you have that old credential synced somewhere that is causing the issue. But check everything, multiple times.
BoomSchtik@reddit (OP)
It's a domain admin, a global admin, Exchange admin, Sharepoint admin. The list goes on. I could totally blow it away, but it would definitely be a pain.
This is not my normal daily driver account.
Ihaveasmallwang@reddit
Why does that account have all those permissions?
Your on prem ad admin account shouldn’t even be touching Entra. It shouldn’t be synced at all.
Create online only accounts for admin functions on M365 and don’t even directly assign roles to it. Use PIM to elevate to the correct role as needed. You shouldn’t even use global admin on your account. You can accomplish 99% of everything you need with another role.
Set it up correctly and then blowing away an on prem admin account is simple. Just create a new account as a copy and delete the old one.
BoomSchtik@reddit (OP)
Yeah... you're not wrong. We'll be fixing the admin accounts to Entra in the reasonably near future.
THe_Quicken@reddit
This. Your Admin account/s should not be “attached” to anything.
Enterprise Admin should only be added to domain admin account as needed then removed. Service accounts for everything. You mentioned PDQ. Should be using laps or at the very least separate service accounts for workstations and servers.
Key-Boat-7519@reddit
You’re chasing a stale credential on that server; find it, then decouple your admin roles. Enable Netlogon logging and audit 4771/4776. On the offender, klist purge, then enumerate anything running as that user: schtasks /query /v | findstr youraccount, Get-Service | where StartName -like, check IIS app pool identities, and remove PDQ stored creds. I use PIM and Windows LAPS for privileged access, with DomainGuard to reduce phishing-driven lockouts. Bottom line: kill the stale cred on that host and keep cloud admin accounts online-only with separate service accounts.
XInsomniacX06@reddit
You should also not be using a password with those accounts you wouldn’t have the lockouts either.besides that You have a process using those creds, on that server. Check services, mapped drives, startup scripts . When the accounts locked something isn’t working . Maybe something old you don’t use anymore. Use procmon to trace what’s happening and look for a process using those creds.
tepitokura@reddit
Exactly this.
phungus1138@reddit
Inside of PDQ there is a section for storing login info for accounts with admin rights. Could be that something is still running even with the services stopped. Should be under Options - Credentials.
BoomSchtik@reddit (OP)
That was the first place I removed my creds when this problem started. It's been deleted out of there for at least 10 days and the lockouts continue.
phungus1138@reddit
Have you checked the sign-in and audit logs in Entra ID? If this is running Azure AD Sync then there may be something there. Perhaps your password doesn't meet complexity requirements or something like that? Just spitballing here.
BoomSchtik@reddit (OP)
It was a good though. I don't see anything that stands out in those two logs for the past 7 days.
phungus1138@reddit
Is your AD sync still working? Could be tied into that.
BoomSchtik@reddit (OP)
yeah... that's working fine.
Gron_Tron@reddit
Are you auditing event ID 4625 on the server you think the lockout is coming from? What does it show for the logon process, status & substatus fields? Anything else useful in there?
SideScroller@reddit
I was looking for this. Does no one check logs anymore?
BoomSchtik@reddit (OP)
I've checked logs. I've written scripts to parse the logs. I've ran tools to report on the logs.
Yes... I have been using the logs.
DeadStockWalking@reddit
I also live in the logs because that's what they are there for.
Wolfram_And_Hart@reddit
No I’ve had to teach all the techs to look their first and work the list.
Run windows store updates folks.
man__i__love__frogs@reddit
Not only that, enable netlogon logging on DCs until you capture one.
forkinthemud@reddit
That's all I do 😭
StormB2@reddit
This - op you need to audit your member server to get the PID of the process causing it.
wildcopper@reddit
I'm not sure if you've already tried this. We all know about cmdkey.exe but try cmdkey.exe as SYSTEM via psexec
C:\Windows\System32>psexec -s cmd
C:\Windows\System32>whoami
nt authority\system
C:\Windows\System32>cmdkey /list
If you see the account listed then go ahead and of course delete it via that cmdkey /delete:targetname...
BoomSchtik@reddit (OP)
I had not tried that! It just showed three generics and the service account for PDQ. Nothing with my creds. Thanks for the suggestion.
I-baLL@reddit
This suggestion is most likely the answer. The question is what machine are you running it on? You have contradictory info in your posts and comments. You say that the lockouts are happening from a server but then you say you’re aware of what machine is causing the lockouts. Where did you run the instructions above on? The machine causing the lockouts? The server? Which one?
Also, when do the lockouts happen? Do they occur when you’re not in the office? If they occur only when you’re in the office then it’s almost definitely cached credentials on a device that you’re carrying.
BoomSchtik@reddit (OP)
The machine causing the lockouts IS the server. I'm using the terms interchangeably. I ran the cmdkey command on the machine/server that the DC logs are saying are causing the lockouts.
I have an export of the lockout events from Splunk for the last 7 days. Reddit won't let me past them in since it's 64 rows. Generally, they start in the early morning between 6 - 8 and continue throughout the day until between about 5 - 8 pm. I get that makes it sound like a scheduled task, but I just can't find a scheduled task that has anything to do with my creds.
RandyCoreyLahey@reddit
does PDQ have any saved accounts running on a specific jobs? like veeam saved creds it wouldnt be on the service but the service would call it
pindevil@reddit
Netlogon debug from the DC where lockout is generated
https://learn.microsoft.com/en-us/troubleshoot/windows-client/windows-security/enable-debug-logging-netlogon-service
BoomSchtik@reddit (OP)
I've enabled netlogon logging, but I'm not seeing anything relevant to do with my account. I see other accounts trying to log in via wireless and failing, but I don't see anything coming from the server in question. I suppose I could be looking at the wrong thing.
kieranken@reddit
This is the way
mauro_oruam@reddit
Does this server host your vpn client ?
Does your vpn require only username and password? Or some type of auto generated code also. Questions to ask your self no need to provide specific info.
Have you checked the logs and make sure it’s not somebody running a dictionary attack against that account?
BoomSchtik@reddit (OP)
Nope, nothing to do with VPN on this server.
reloadtak@reddit
And you are 100% sure this server does not have the NPS role installed?
BoomSchtik@reddit (OP)
This server doesn't have anything to do with RADIUS directly.
firedocter@reddit
Check credential manager.
BoomSchtik@reddit (OP)
Credential manager has been cleared previously (for my account and SYSTEM). It has not helped.
whistlepete@reddit
We recently had a similar issue where a user kept getting locked out and it was hard to track down. Turns out they had logged in as a local account on the machine but then mapped a network drive with their ad creds. It was hard to find because the mapped drive only showed up under the context of that local login.
Do you have any mapped drives under any other accounts in that server?
BoomSchtik@reddit (OP)
No, I don't believe that mapped drives are a factor here. I can log into the server and the DC with the admin account and there aren't any drives mapped, confirmed with 'net use'.
che-che-chester@reddit
When I can't figure out if an account is being used for a task, service (unlikely), RDP, mapped drive, etc. but I know it is authenticating on a certain machine, I'll try FullEventLogView from NirSoft. It takes the 100+ event logs and views them all in one window. Get the exact time of a bad password event, then use this tool to look at the 5-10 seconds before that event. I would also make sure you have process creation auditing enabled.
https://www.nirsoft.net/utils/full_event_log_view.html
BoomSchtik@reddit (OP)
I hear you. I only used my creds in PDQ for machines that had not yet had their admin groups modified. Once the admin group is modified, then the PDQ service account can be used from that point forward. Sometimes you just need domain admin rights for stuff. Fortunately, my boss does not feel like you do.
I_turned_it_off@reddit
Was this account used to set up the Azure AD sync? if so it could be retaining the authentication ticket from before you last changed your password, try running through the configuration again and see if that resolves the issue.
BoomSchtik@reddit (OP)
I created an Enterprise Admin service account for Microsoft Entra Connect Sync to use to try to eliminate the possibility of it coming from that.
jonstafari@reddit
not sure if this is sorted out for you at all, but I had a very very similar issue when I started at a new company. For contenxt....The company ticket volume ended up having 30% locked accounts going to service desk. Wasted tons of time on the user and eng side.
I enabled verbose logging on the netlogon services on our domain controllers, and set up a simple powershell scheduled task to do the log rotation. I also used the netwrix account lockout tool as well, but in the end, you need to know 2 things: domain controller(s) tracking the locked account, and the verbositiy of the logs to show:
Account Name (which I think it does by default w/o verbose)
Caller Computer Name (verbose).
When the account is locked, you should be able to see which DC picked it up, and then review the netlogon file to confirm the computer name/details as well.
If you can, tie in your Observabily or Monitoring tool to alert when a domain controller picks up the eventID of 4740 in the Security Logs -- that's a locked account.
Good luck mate!
wrootlt@reddit
Perplexing
BoomSchtik@reddit (OP)
Indeed!
sausages20@reddit
My two cents, you have already done a lot of troubleshooting and sounds like the apps you expect are still working.
Things I haven’t seen you check: - Security event log on the server in question to check if coming from a different source - Netlogon debug logs on dc to see if it is coming via a different machine (although I don’t think it is this) - Run procmon / wireshark to look for processes in your name and then correlate to lockout event to trace back to specific app or something. - Manually mapped network drive marked as persistent?
Procmon you should be able to filter to your username to look for things launching as you but may not catch it super well as it can be a tad busy. Wireshark look for smb2, kerberos, dcerpc, ldap. Find an account username event when you get an event and right click and add to the columns at the top.
timwtingle@reddit
Change the password back to stop the bleeding. Create new service accounts for each individual service and make it a descriptive name. Only use those for one service. Perhaps change the admin password again (why though) and see if you can isolate the cause.
Bulky_Somewhere_6082@reddit
One item I didn't see in this chat was anything about a Scheduled Task that runs with your creds. Did you set up one of those cause if so, it has your old password and will lock the account every time it runs.
BoomSchtik@reddit (OP)
That's bullet point number 4
myutnybrtve@reddit
I changed my username when this happened to me. Annoying bad solution. But it worked.
jasonsyko@reddit
Cached creds/kerberos.
Have you cleared the creds and cleared Kerberos tickets and log off/back on?
BoomSchtik@reddit (OP)
Cleared credential manager yes. I just cleared Kerberos tonight, so we'll see if that helps.
AmoebaAffectionate71@reddit
Is your admin account a member of protected user group in AD?
BoomSchtik@reddit (OP)
It is not a member of Protected Users
Swordbreaker86@reddit
Kill the account and make a new one? Some stuff isn't worth chasing after x amount of effort.
BoomSchtik@reddit (OP)
This account is deeply rooted in the administration of the domain. That would definitely be a worst-case scenario, but I won't totally rule it out.
entyfresh@reddit
This is the real issue you should be seeking to fix right now
Ihaveasmallwang@reddit
It shouldn’t be. Take this as an opportunity to set it up correctly, even though it’s a pain to get there.
totally_not_a_bot__@reddit
then you might end up with two broken accounts.
I'd rebuild the server before doing that.
iccccceman@reddit
Any mobile devices or anything you've logged in to? I remember this happening and it was related to a device that was hitting AD with old credentials. That would line up with your password change too.
ScarcityReal5399@reddit
Silly question, but did you test the PDQ deploy and inventory credentials? Sounds like an auto deployment that is using the old credentials to run
BoomSchtik@reddit (OP)
I removed my creds out of options -> credentials as soon as this started. Then I stopped the service and set it to disabled, so it SHOULDN'T be a factor in what's going on.
CPAtech@reddit
You need to shut down the server to be sure.
BoomSchtik@reddit (OP)
Server's been rebooted multiple times with the services disables so that PDQ can't start back up.
CPAtech@reddit
That's not the same as shutting it down completely and attempting to reproduce the lockouts.
Ws6_@reddit
I had a similar situation but with all the domain accounts. It turned out the the FortiGate was being brute forced on our SSL VPN port. The MFA setup on the FW queries the DC, and that was being hit hard. I hope that helps
hardingd@reddit
Why has no one suggested using procmon or process explorer from Sysinternals. There are YouTube videos that will show you how to find the process that is failing authentication
Brad_from_Wisconsin@reddit
I majored in system reboots in tech school.
Could the server be logged in as the account the that is getting locked. For example you changed the password on a different system but did not log out on this computer since you did it. I know it is probably a long shot but it will give you a chance to take a short walk around the building while the server is rebooting and you might come up with a better idea while you are taking a walk.
And who knows rebooting the server might just solve the problem.
BoomSchtik@reddit (OP)
I've rebooted it multiple times. The affected account can log into the offending server no problem.
Brad_from_Wisconsin@reddit
which of the services that were identified by Perplexity are unable to be stopped?
maggotses@reddit
Azure AD Sync has credentials embedded in the setup.
Turbojelly@reddit
There is a computer somewhere running a script against the server with an old admin password hardcoded in it.
spydum@reddit
this is likely it: while this server is the one being reported as locking it out -- thats only because this server runs the service. There is a script calling some service on here with the old cred. Normally the 4625 event messages tell you what process failed the login?
Working_Average_2789@reddit
I just tossing this around that might help you but could it be the AzureADKerberos object in your AD that acts as a RODC. A solution could be to rotate the key Link
I hope you will updated the post if you find the solution to this.
jivatma@reddit
Did you setup a network share using your admin account somewhere?
BoomSchtik@reddit (OP)
Not that I can think of. The Netwrix tool showed me where I had inactive rdp sessions on servers, so I was able to go and close those out. We don't use mapped drives much.
MartinDamged@reddit
Doesn't need to be mapped to a drive letter.
It could just be a network share mount.
CraigAT@reddit
Look at all the applications running on that server, including scheduled tasks etc. Any you find using your admin credentials, change to use new service accounts, document these securely in a password vault. Even if this doesn't find the issue, it puts you in a better place.
Are the lockouts or password attempts regular? Maybe run process explorer to see what is running on the server?
LUXCADE@reddit
Use EventCombMT to find where's the source of lockout. Select account lockout option and add the event ID 4740 Run and wait for the results.
aaiceman@reddit
Change the username?
Rhythm_Killer@reddit
Come on, that’s basically giving up!
BoomSchtik@reddit (OP)
It's an option if I get desperate
bit0n@reddit
Username change seems the answer the set up a test account with the old username to keep investigating.
Last time I had this it was our VPN being probed and the watchguard ad sync tool was spamming requests and causing lockout.
Unexpected_Cranberry@reddit
It's been a few years since I troubleshot this, but I believe you might be misled by the DC log.
If you map a drive on your laptop to the pdq server, the authentication event on the DC will be from the pdq machine, not your laptop.
Have you mapped a share in the pdq from some machine somewhere? Perhaps one you did troubleshooting on recently? Or your own machine if your assigned in with a different account there for day to day stuff.
christheitguy@reddit
Do you by chance use AD Credentials for WiFi?
BoomSchtik@reddit (OP)
We do via RADIUS, but I've had lockouts when my computer was logged out and shut down, so I don't think that part of the issue. I wouldn't use my creds for wifi on any computer other than my own.
slav3269@reddit
Worth eliminating though. See if you can shut down RADIUS and whether lockouts continue
hsod100@reddit
So when you unlock the account what is the exact pattern of the re-lockout occurring. How long, when, is it always the same timing, etc. you say you'll see overnight, so I take it the lockout does not occur immediately. Could be a hint. Scrutiny to system and app logs to cross reference that point in time. Lots of good suggestions already given. Check printers, print queues, and any other devices added.
OneStandardCandle@reddit
If you use Defender, offboard and onboard the server. Mssense has a password protection feature that can hose you. It was patched in June, but needs an off/onboard to fix itself.
BoomSchtik@reddit (OP)
We use SentinelOne.
smc0881@reddit
Do you have Deep Visibility with S1? You might be able to search through the telemetry in there if it's enabled.
2clipchris@reddit
Is your admin cached in on desktop or laptop or a phone connected to WiFi? Assuming it is using authentication methods.
BoomSchtik@reddit (OP)
No, I use my normal creds to connect to wifi via RADIUS.
Beefcrustycurtains@reddit
If all else fails, You could always change your username. Do you have pass through authentication enabled and that account synced with o365? With that combination and that being a PTA server could come from your o365 either brute force or even failed activesync creds, but if it's your admin account I would guess you don't have any mobile mail clients setup.
BoomSchtik@reddit (OP)
We are federated with Okta and most authentication goes through there, but our AD is synced with AAD Sync and this admin account is synched.
I don't use mail with that account.
AmoebaAffectionate71@reddit
Double check your AAD sync configuration. If I’m not mistaken you might have used the admin credentials to read / connect to AD you might have your old admin account credentials there. Run the configuration again and check which accounts were used.
Check other applications as well eg some backup software might have admin accounts saved if you didn’t configure a dedicated accounts for the apps.
SwooshRoc@reddit
Did you use that account at all setting up okta?
BoomSchtik@reddit (OP)
No, Okta was set up before my time. I'm not aware that this account is associated with the way that Okta is working in any way.
Lord-Raikage@reddit
There's a service or program on that dc that is still holding onto your credentials even if you think you deleted them all. This has happened to me before. You need to figure out the services and programs you use on this server daily and 1 by 1 when you access them, save fake credentials to it. They can only hold onto the most recent one so it will eventually release your old password. Good luck
Panx-Tanx@reddit
People are suggesting a lot of information. My 2 cents - These lockouts can also happen, if you have applied cis benchmark on the DC. and something is using NTLMv1. You may want to check if anything like this was done on the DC or a new Domain controllers policy was applied to change the LMComoatibility level to 5.
_Robert_Pulson@reddit
Have you deleted the local profile from the server? Might be a mapped drive, shared folder, COM, web browser, cached credentials, .OST, service account, scheduled task, etc...
Honestly, I'd delete all local profiles from every server/endpoint, and see if it still locks. Otherwise, create a secondary user account.
zrad603@reddit
is the system time/clock correct?
BoomSchtik@reddit (OP)
Yep!
RedditDon3@reddit
Have you kept track of when the lockouts happen? Any particular trends there?
I have a scheduled task running on my PDC to email me the associated event when a lockout occurs. But you’ve already mentioned that you know exactly where the lockouts are coming from, so this is probably not going to help.
TheJollyHermit@reddit
Have you made sure the time is in sync with the domain on the host and all vm? Time drift can cause account lockouts too
BoomSchtik@reddit (OP)
Time looks great.
AveyBleh@reddit
One dumb way I’ve locked out my admin account is by stale connections to $ shares on my workstation from prior to PW change.
Run net use to check for connections then net use \server\share\path /delete to kill it.
A reboot would also kill it.
BoomSchtik@reddit (OP)
nothing on net use on my computer or on the server. There's been a bunch of reboots.
External-Housing4289@reddit
Confirm by powering off said server, lockout stop happening?
SifferBTW@reddit
Why are you using your admin account for services and automation?
Those should be separate accounts.
Get on the device you think is the culprit and start looking at event viewer. Don't just look at failed auth, check services, scheduled tasks, etc.
BoomSchtik@reddit (OP)
I'm not, that was part of the point of my post. There are no services or tasks using my creds, which is lending to the mystery of where the lockout is coming from.
AffectionateMix3146@reddit
Beyond enabling the audit logging as others have mentioned, do you have an EDR tool? Since you know the source machine I assume you have a timestamp you could cross reference with the machine’s telemetry to find the root cause process and go from there
BoomSchtik@reddit (OP)
We use SentinelOne for that. I'm not sure how I'd use it to assist with this.
InfosecPenguin@reddit
If you have deep visibility within S1 you might be able to identify what process is attempting to run as the user in question on that system.
DmetaNextWeek@reddit
This happened to me, and I never bothered to fix it until this week.
Search Security Events in event viewer for event 4625. This was svchost.exe. Cross reference that timestamp with the Event Viewer log for Scheduled Task to show which event was failing at the same time the bad password was being attempted, killed the task and everything stopped locking.
Hope that’s all it is!
BoomSchtik@reddit (OP)
I've checked and rechecked the task scheduler (visually and with PowerShell). There's not a task running with my creds. Was the one in your case running with your creds or showed something else?
DmetaNextWeek@reddit
It wasn’t, it was a script erroring out run by task scheduler that I was able to find based on the timing of the bad password security event
TylerJurgens@reddit
Veeam guest processing (if you're using Veeam)?
BoomSchtik@reddit (OP)
No Veeam.
andibogard@reddit
Application pool in IIS?
BoomSchtik@reddit (OP)
No IIS on this server. I did think of that at some point.
briskik@reddit
My best guess is ad Azure sync, and possibly using this account to do the initial connection - and it misbehaving with a cached cred. Can you reinstall it, or rerun that wizard?
BoomSchtik@reddit (OP)
I created an Enterprise Admin service account for Microsoft Entra Connect Sync to use to try to eliminate the possibility of it coming from that.
briskik@reddit
Also there is some regedit key to enable that allows more logging to the security event viewer on this server, to see events, then filter for your account
buckers13@reddit
What do the logs say?
maggotses@reddit
Phone or tablet with account linked?
BoomSchtik@reddit (OP)
It would only be my phone and Teams and Outlook are working fine on it.
IT_vet@reddit
Proxy creds maybe? Sounds crazy, but I’ve done it on a machine that was not part of the company domain but needed to auth through our web proxy.
BoomSchtik@reddit (OP)
I don't think so. We don't really use any proxies internally.
FatalSky@reddit
Oh I’ve dealt with that shit. Mine was an account used for file sharing. All the clients had old cached auto login credentials to the account and kept triggering account lockout.
Natural_Sherbert_391@reddit
I hate to quote AI, but I gave Gemini your post. Too long to post the whole thing but here are a couple of suggestions:
WMI/COM Objects: This is a very common blind spot. Some applications, especially older ones, use Windows Management Instrumentation (WMI) or COM/DCOM to call remote services, and they might have cached or hardcoded credentials. Action: Open DCOMCNFG (dcomcnfg.exe) and check through the list for custom applications. Look at the properties for any custom services to see if a specific user identity is assigned.
IIS Application Pools (If Installed): If this server runs any web services (even for PDQ), check the Identity of all Application Pools in Internet Information Services (IIS) Manager.
Azure AD Connect (AADC) The AADC service uses a managed service account by default, but there are a few places where you might have overridden it with your admin credentials during setup: Connector Credentials: Check the Synchronization Service Manager tool. Go to Connectors, right-click the Active Directory Connector, select Properties, and look under Connect to Active Directory Forest. Ensure the "Connect Using" option is not set to an account using your admin credentials. PDQ Deploy/Inventory (Even when stopped) You said the services are stopped, but sometimes the background scanning/heartbeat process of Inventory can try to "wake up" and fail, or there could be a persistent connection. Check the Database: If PDQ is using an internal or external database, it might have a connection string that uses your credentials. Check the PDQ options. Clearing PDQ Credential Cache: PDQ stores credentials for different targets. While the services are stopped, this shouldn't be the issue, but as a last resort, check the configuration files for any residual credential bloat.
. The Nuclear Option (To Isolate the Process) If all else fails, you need to use a tool to catch the attempt in the act, or take the server off the network. Process Monitor (ProcMon) with Filtering: Download and run Process Monitor (ProcMon) from Sysinternals. Set a Filter to include only Process Operation is LsaLogonUser or Result is ACCESS DENIED. This is a low-level operation that will catch the process trying to authenticate. Wait for the lockout to occur and check the ProcMon trace to see which Process Path initiated the failed login attempt.
flexcabana21@reddit
Find the account locked out using event id 4740 in Event Viewer
marklein@reddit
Fuck it, it's an admin account. Disable it and make a new one.
raip@reddit
Turn on failed login auditing and check the event logs. It'll tell you what client is driving the failed logon.
BoomSchtik@reddit (OP)
It is on and I do know which machine the lockouts are originating from.
sitesurfer253@reddit
Credential manager on the machine the lockouts come from?
raip@reddit
Cool, check the security log on that computer (if it's Windows), it'll tell you the PID (in hex format) of the process that failed to logon.
Br3m3n@reddit
Since you mention Network have you looked at the failed login activity report for that account. Look for a server that may have cached credentials that you have connected to since the password change.
blow_slogan@reddit
You should be able to see in the logs which workstation or IP the bad authentication attempts came from.
Acceptable-Wind-7332@reddit
Have you used your admin account as a service account, the some time later changed the password? If so, update the password in services.msc or get a separate service account.
BoomSchtik@reddit (OP)
There are no services running with my creds. I've double and triple checked that.
NorthAntarcticSysadm@reddit
As weird as this may sound, what about lighting up a new DC and temporarily shutting off the DC where the problem is happening?
Have you also tried enable login auditing on all endpoints and using a SIEM, XDR or log aggregation tool to see if the creds are on a workstation?
Is the account potentially authenticating on a VPN or gateway somewhere remotely? Maybe VPN client on an old device?
Another thought, you entered the credentials somewhere (end point device) as a trial or bandaid forgot to go back to it.