My first MSP had the same local admin password on every customers machines. 8 chars.Then I learned what LAPS was.

Posted by hardeningbrief@reddit | sysadmin | View on Reddit | 30 comments

My first real IT job was at a small MSP, maybe forty customers across a mix of industries, a few thousand endpoints between them, and in my second week I was shadowing one of the senior techs on a ticket when he casually RDP'd into a client workstation and typed the local admin password in from memory, which at the time struck me as completely normal because every experienced tech I'd ever watched seemed to have that kind of muscle memory for credentials, and I figured this was just what being good at the job looked like.

The next ticket was a different customer, different city, different industry entirely, and he typed the same password again, and I genuinely did not notice, because why would I, I was new and trying to keep up with ticket flow and learning the PSA and figuring out which customer was which, and the idea that two unrelated companies would share a local admin password wasn't something my brain was set up to even register as weird yet tbh.

It took me a few more weeks (or months, it's been a couple years) before I actually clocked what was happening, and when I did I remember kind of laughing about it, because the whole thing was so matter of fact: every managed endpoint across every customer we had was using the exact same local admin password, it had been that way for something like four years, it was stored in the shared KeePass, every tech learned it on day one, and every tech typed it ten times a day until it lived in their fingers, and when I asked one of the seniors about it he shrugged and said something along the lines of "it's a local account, it's not domain-joined admin, it doesn't really matter, we've always done it this way and nothing has ever happened."

And I accepted that, because he had been in the industry for fifteen years and I had been in it for three weeks, and the customers weren't complaining, and the MSP wasn't getting hit with anything visible, and life went on.

I want to be honest about this because I think a lot of people reading this have either lived the same setup or are living it right now: at no point in that first year did anyone, including me, treat it as an actual risk. It was just the way the shop worked. Customer onboarding meant imaging the machines, setting the local admin to the shared password, joining them to the domain, and moving on to the next ticket. We weren't being reckless on purpose, we were operating on the assumption that local admin wasn't that important because it "wasn't domain," and nobody in that office had the time or the incentive to sit down and actually trace what a local admin compromise would look like in practice.

What I only understood much later, and what I wish I'd understood on day one, is the actual attack path this setup hands to anyone who gets even a foothold on one machine.

somebody phishes a user or exploits an unpatched CVE or walks off with a stolen laptop, and they end up with SYSTEM on that one endpoint, which is bad but not catastrophic if the environment is segmented properly, and then they dump the local SAM with secretsdump or mimikatz or whatever flavour of tooling is current, and they pull the local admin NTLM hash out of it.

In a properly configured environment that hash is useless anywhere else because every machine has its own unique password and therefore its own unique hash, but in an environment like ours it was the master key, because every single machine across every single customer had the same local admin hash, which meant pass-the-hash against any other box in the subnet worked on the first try, and from there it was just a matter of finding one machine with a cached domain admin token sitting in memory from a recent session, which in an MSP environment where admins move between boxes constantly is basically a guarantee. One compromised endpoint becomes every endpoint, every endpoint becomes the domain, the domain becomes the customer, and if your tooling is shared across customers the way MSP tooling usually is, now we're having a much worse conversation.

I only found out about LAPS maybe six or seven months into the job, reading something unrelated on a lunch break, and I remember the feeling of it landing because it was the first time I realised that Microsoft had built a native fix for this exact problem years earlier and we had just... not deployed it. When I brought it up internally the response was that it was "too much work to roll out across all the tenants," which in retrospect is the single sentence that tells you everything you need to know about why MSPs end up in the news for ransomware incidents. It wasn't malice, it wasn't even really negligence in the legal sense, it was just the default human thing of deprioritising work that doesn't have an obvious deadline attached to it.

I left that job a while back, and I've spent enough time since then talking to other sysadmins and MSP engineers at conferences and meetups to know that this pattern is still extremely common, either in its original form (shared local admin everywhere, no rotation, no auditing) or in a slightly more polished form where somebody deployed LAPS but did it wrong, usually by making Domain Admins the password reader group and calling it done, which defeats the point of the tiering model the whole thing is supposed to enable.

So I wrote up a proper walkthrough recently( this is not self promo, plz have mercy mods....) because every time I explain this to someone new at my current company I end up repeating the same six steps, and I figured I'd rather just have them in one place. The short version for anyone who wants the gist before deciding whether to click anything is that legacy Microsoft LAPS (the old MSI that shipped back in 2016) is deprecated and blocked on Windows 11 23H2 and later, so if you're still running it you need to migrate using the Legacy Emulation Mode that Microsoft built into Windows LAPS specifically for this purpose, and Windows LAPS itself has been native to every supported Windows OS since the April 2023 cumulative update, meaning no MSI, no agent, no licensing cost, no reboots on clients.

The MVP deployment is roughly thirty minutes of work and involves verifying the LAPS PowerShell module on a DC, running Update-LapsADSchema once on the Schema Master to extend the forest schema, granting computer self-permission on your target OU with Set-LapsADComputerSelfPermission, enabling auditing with Set-LapsADAuditing, configuring the GPO under Computer Configuration → Policies → Administrative Templates → System → LAPS with the backup directory, complexity, length (20 minimum), rotation age (30 days), and post-authentication actions (reset and logoff, 24h grace), and then verifying retrieval with Get-LapsADPassword.

The part most people get wrong, and the part that matters more than the deployment itself, is the reader group configuration. Do not use Domain Admins as your LAPS password reader group, because the entire point of LAPS in a tiered environment is that helpdesk can read workstation passwords without being able to read DC passwords, and if Domain Admins can read everything then you haven't actually separated anything. Create dedicated per-tier reader groups (something like LAPSPasswordReaders-T1 and LAPSPasswordReaders-T2), scope each one to the OUs at the correct tier, and on T0 systems you should be disabling the RID-500 admin account entirely rather than managing it. The difference between an environment that "has LAPS" and an environment that has LAPS correctly is almost entirely in this step.

For detection you want to be forwarding two event IDs from the Windows LAPS Operational channel (Applications and Services Logs → Microsoft → Windows → LAPS → Operational) to your WEC or SIEM: 10022 fires when a password is read and is your tripwire for unauthorised access, and 10026 fires when rotation fails and should be investigated same-day because a machine that isn't rotating is a machine that's drifting back toward the original problem.

If you're at an MSP right now reading this and the story at the top feels familiar, the excuse that it's too much work to roll out across your tenants stops being true the moment you've done it once and have the GPO template saved, and you're trading maybe thirty minutes per tenant against the ransomware incident that could genuinely end the business.

Happy to answer anything in the comments.