My first MSP had the same local admin password on every customers machines. 8 chars.Then I learned what LAPS was.
Posted by hardeningbrief@reddit | sysadmin | View on Reddit | 30 comments
My first real IT job was at a small MSP, maybe forty customers across a mix of industries, a few thousand endpoints between them, and in my second week I was shadowing one of the senior techs on a ticket when he casually RDP'd into a client workstation and typed the local admin password in from memory, which at the time struck me as completely normal because every experienced tech I'd ever watched seemed to have that kind of muscle memory for credentials, and I figured this was just what being good at the job looked like.
The next ticket was a different customer, different city, different industry entirely, and he typed the same password again, and I genuinely did not notice, because why would I, I was new and trying to keep up with ticket flow and learning the PSA and figuring out which customer was which, and the idea that two unrelated companies would share a local admin password wasn't something my brain was set up to even register as weird yet tbh.
It took me a few more weeks (or months, it's been a couple years) before I actually clocked what was happening, and when I did I remember kind of laughing about it, because the whole thing was so matter of fact: every managed endpoint across every customer we had was using the exact same local admin password, it had been that way for something like four years, it was stored in the shared KeePass, every tech learned it on day one, and every tech typed it ten times a day until it lived in their fingers, and when I asked one of the seniors about it he shrugged and said something along the lines of "it's a local account, it's not domain-joined admin, it doesn't really matter, we've always done it this way and nothing has ever happened."
And I accepted that, because he had been in the industry for fifteen years and I had been in it for three weeks, and the customers weren't complaining, and the MSP wasn't getting hit with anything visible, and life went on.
I want to be honest about this because I think a lot of people reading this have either lived the same setup or are living it right now: at no point in that first year did anyone, including me, treat it as an actual risk. It was just the way the shop worked. Customer onboarding meant imaging the machines, setting the local admin to the shared password, joining them to the domain, and moving on to the next ticket. We weren't being reckless on purpose, we were operating on the assumption that local admin wasn't that important because it "wasn't domain," and nobody in that office had the time or the incentive to sit down and actually trace what a local admin compromise would look like in practice.
What I only understood much later, and what I wish I'd understood on day one, is the actual attack path this setup hands to anyone who gets even a foothold on one machine.
somebody phishes a user or exploits an unpatched CVE or walks off with a stolen laptop, and they end up with SYSTEM on that one endpoint, which is bad but not catastrophic if the environment is segmented properly, and then they dump the local SAM with secretsdump or mimikatz or whatever flavour of tooling is current, and they pull the local admin NTLM hash out of it.
In a properly configured environment that hash is useless anywhere else because every machine has its own unique password and therefore its own unique hash, but in an environment like ours it was the master key, because every single machine across every single customer had the same local admin hash, which meant pass-the-hash against any other box in the subnet worked on the first try, and from there it was just a matter of finding one machine with a cached domain admin token sitting in memory from a recent session, which in an MSP environment where admins move between boxes constantly is basically a guarantee. One compromised endpoint becomes every endpoint, every endpoint becomes the domain, the domain becomes the customer, and if your tooling is shared across customers the way MSP tooling usually is, now we're having a much worse conversation.
I only found out about LAPS maybe six or seven months into the job, reading something unrelated on a lunch break, and I remember the feeling of it landing because it was the first time I realised that Microsoft had built a native fix for this exact problem years earlier and we had just... not deployed it. When I brought it up internally the response was that it was "too much work to roll out across all the tenants," which in retrospect is the single sentence that tells you everything you need to know about why MSPs end up in the news for ransomware incidents. It wasn't malice, it wasn't even really negligence in the legal sense, it was just the default human thing of deprioritising work that doesn't have an obvious deadline attached to it.
I left that job a while back, and I've spent enough time since then talking to other sysadmins and MSP engineers at conferences and meetups to know that this pattern is still extremely common, either in its original form (shared local admin everywhere, no rotation, no auditing) or in a slightly more polished form where somebody deployed LAPS but did it wrong, usually by making Domain Admins the password reader group and calling it done, which defeats the point of the tiering model the whole thing is supposed to enable.
So I wrote up a proper walkthrough recently( this is not self promo, plz have mercy mods....) because every time I explain this to someone new at my current company I end up repeating the same six steps, and I figured I'd rather just have them in one place. The short version for anyone who wants the gist before deciding whether to click anything is that legacy Microsoft LAPS (the old MSI that shipped back in 2016) is deprecated and blocked on Windows 11 23H2 and later, so if you're still running it you need to migrate using the Legacy Emulation Mode that Microsoft built into Windows LAPS specifically for this purpose, and Windows LAPS itself has been native to every supported Windows OS since the April 2023 cumulative update, meaning no MSI, no agent, no licensing cost, no reboots on clients.
The MVP deployment is roughly thirty minutes of work and involves verifying the LAPS PowerShell module on a DC, running Update-LapsADSchema once on the Schema Master to extend the forest schema, granting computer self-permission on your target OU with Set-LapsADComputerSelfPermission, enabling auditing with Set-LapsADAuditing, configuring the GPO under Computer Configuration → Policies → Administrative Templates → System → LAPS with the backup directory, complexity, length (20 minimum), rotation age (30 days), and post-authentication actions (reset and logoff, 24h grace), and then verifying retrieval with Get-LapsADPassword.
The part most people get wrong, and the part that matters more than the deployment itself, is the reader group configuration. Do not use Domain Admins as your LAPS password reader group, because the entire point of LAPS in a tiered environment is that helpdesk can read workstation passwords without being able to read DC passwords, and if Domain Admins can read everything then you haven't actually separated anything. Create dedicated per-tier reader groups (something like LAPSPasswordReaders-T1 and LAPSPasswordReaders-T2), scope each one to the OUs at the correct tier, and on T0 systems you should be disabling the RID-500 admin account entirely rather than managing it. The difference between an environment that "has LAPS" and an environment that has LAPS correctly is almost entirely in this step.
For detection you want to be forwarding two event IDs from the Windows LAPS Operational channel (Applications and Services Logs → Microsoft → Windows → LAPS → Operational) to your WEC or SIEM: 10022 fires when a password is read and is your tripwire for unauthorised access, and 10026 fires when rotation fails and should be investigated same-day because a machine that isn't rotating is a machine that's drifting back toward the original problem.
If you're at an MSP right now reading this and the story at the top feels familiar, the excuse that it's too much work to roll out across your tenants stops being true the moment you've done it once and have the GPO template saved, and you're trading maybe thirty minutes per tenant against the ransomware incident that could genuinely end the business.
Happy to answer anything in the comments.
aes_gcm@reddit
This is 100% AI, no one types like this.
ZanshinMindState@reddit
So many of these posts here now that it’s disrupting the usefulness of the subreddit…
WDWKamala@reddit
Thanks ChatGPT
Flaky-Gear-1370@reddit
Like holy shit it’s so obvious and then the posts responding that are also ChatGPT 😂
Evening-Page-9737@reddit
What are they even gaining by doing this? Maybe I've missed something but I don't even understand what the point of dead interneting reddit has been in the first place. AI bots responding to AI bots on a forum platform is so useless.
OneSeaworthiness7768@reddit
Looks like they run a blog or paid newsletter and that’s what they’re advertising/doing SEO for.
aes_gcm@reddit
As far as I know, it's for promotion. They generate a post like this with AI, people upvote it because its interesting, it's highly visible on the subreddit and in aggregate feeds because of the engagement, and then the same user (or another account that's also controlled by the same person) suggests a paid tool or drops a URL.
Excellent-Program333@reddit
I am waiting on the sales pitch of a tool that is amazing!
Vodor1@reddit
You're absolutely right! Get LAPS, It's Amazing!
Wait, let me edit that to put a bullet point in first.
GoodCrossing@reddit
This sub is honestly so full of blatant LLM-generated posts. Some are clearly people that want to have an LLM "fix up" the wording, so at least you know there's an actual human wanting to post whatever it is. But most are some 1 month old account with an AI posting a bunch of comments and posts about one topic and then they magically start selling some tool to fix their AD/backup solution/provisioning/whatever vibecoded BS they want to sell.
This user's post history is all tutorials about AD security, which they started posting right as they made their account! Oh and of course he has a pinned post in the user page for their... AD hardening website.
(emphasis mine)
jfoust2@reddit
Come now, this wordy and competent sysadmin has been posting for almost a month now!
gmerideth@reddit
I mean I always use arrows → when → responding to comments or posting → new topics. Don't you guys?
magataga@reddit
I assure you my friend, having a password manager with the ability to generate arbitrary passwords for each client and instead choosing to use a shared 8 character password does constitute legal negligence.
This violates the most basic security tenant of non-repudiation, as well as basic requirements for due diligence and due care.
Conscious-Arm-6298@reddit
You know how many enterprises still have the root user active in most of the infra ? And I'm talking a non local IT fault, this is primarily enforced by the third party service providers, it's crazy.
I-D3STRUCTI0N-I@reddit
Why wouldn't you have a root user?
Conscious-Arm-6298@reddit
Its a common security practice to disable the default root user since it's common for all systems , making it the primary attack target. I'm not saying to not have users with root privileges, as it's of course, needed.
Regen89@reddit
Had to take a break here 🤣. I feel like this is close to administration 101, maybe out of the mouth of a t1/2 tech that doesn't know better, but senior staff that is training you.. yeah MSPs I guess.
Not really what you ever want to hear from someone you've handed the keys to the kingdom to. At least Indian MSPs will lie to you and try to hide their wrongdoings instead of just admitting they are reckless and don't know what they are doing.
You might be surprised what the legal standard for negligence can be. Imagine one of your customers got skullfucked and with a lawyer in the room you repeated the words 'we weren't being reckless on purpose'. 😂
Ok I've reached my limit for responding to bots for the day o7
No-Land-672@reddit
At a client’s IT environment, which had been set up by our predecessor, LAPS was installed. The only Domain Controller in the admittedly very small network went down, so initially it was not possible to log in on the clients with administrative privileges, neither locally nor via the domain. I still remember that we were able to reset the local passwords using Linux-based tools. At that time, the main goal was simply to get the clients online quickly and to make software temporarely operational that did not depend on the Domain Controller.
I am generally in favor of LAPS, but only if at least two Domain Controllers are available in the network, to ensure redundancy in case one goes down.
joshbudde@reddit
Its a contentious opinion, but I'm of the opinion that for any network under 100 endpoints, AD is unnecessary and just a way to cut fat checks to MSPs.
Glass_Call982@reddit
Yep. And most small orgs won't shell out the money for another server license. Which I don't understand. If you can't afford that how are you even profitable or making payroll?!
Previous-Low4715@reddit
I deploy LAPS via Intune and a specific PIM role elevation is required to pull the password for each machine. We have no local admin accounts besides the LAPS account which self destructs after it's used. It's fantastic, we reduced points of lateral movement by hundreds of thousands. Service Desk didn't like it of course, but they got used to it.
mkfelidae@reddit
As a service desk tech. This wouldn’t bother me at all provided three things: 1) the timeframe increase is factored into my KPI’s 2) those KPI’s are used to determine if we need another technician based on that change 3) the change is communicated over a reasonable timeframe, not at 4pm on a Thursday, in effect at 6pm the same day
DiscipleOfYeshua@reddit
Same. This is the way. (For now).
Ironic_Jedi@reddit
Same here. Deployed it via intune policy with the backing of my awesome cyber team and service desk was annoyed at first but now used it.
Really easy to implement and on Windows 11 you can even have the policy create the local account so you don't have to use the built-in administrator account.
hardeningbrief@reddit (OP)
Thats even better! Nice way to do it.
And yes, admins and users hate the friction security brings.. but it's what keeps us our jobs ;)
aardvark_xray@reddit
A certain medical device manufacturer (who also makes airplane engines) learned this the hard way too. Imbedded universal/global accounts, no matter how limited, are never a good practice.
Global admin accounts are really bad practice.
Opposite_Bag_7434@reddit
This is excellent OP!
This is not just a problem with many MSP’s but also a pretty common issue in many smaller enterprises as well. I’ve joined a few that were using common shared administrative credentials.
Funny story, an HR manager during my orientation at one company where I was hired to lead an existing IT team. Pretty sizable company too. Anyway, it was clear during that orientation that they had some very dodgy security practices. I distinctly remember mentioning that this wasn’t good and that it was going to change out loud and the HR manager noticed.
Like your MSP every remote site (around 100 of them) and every device had the same admin account.
The issue is pretty widespread. Surprisingly so is the lack of endpoint security, but that’s an entirely different nightmare completely.
MeetJoan@reddit
The reader group misconfiguration is the one that gets glossed over in every LAPS write-up I've seen, so glad you called it out specifically. "We have LAPS" and "LAPS is actually doing what it's supposed to" are two very different things, and the gap between them is almost always that someone made Domain Admins the reader group and considered the job done.
The post-authentication actions setting (reset and logoff after 24h) is also underdeployed. Most environments I've seen have LAPS configured but no PAA, which means a compromised password stays valid until the next scheduled rotation. Not ideal if someone is quietly using it.
Good write-up. The migration path from legacy LAPS via emulation mode is the part most people don't know exists and will save a lot of pain.
hardeningbrief@reddit (OP)
Thank you! Appreciate it :)
beneschk@reddit
This is me anytime I join an MSP and they're attempting to patch windows workstations in their RMM.
Your RMM is not servicing stack aware. It doesn't understand the order of KBs/cumulatives which inevitably causes disk bloat in the winSxS and a corrupt component store due to pending KB installs getting stuck because the last cumulative already applied it.
Its also the same thing when trying to explain Windows 10+ has driver signature enforcement. Meaning weird issues will pop up if you have the incorrect drivers for that build (VPN stops working/intermittent wifi disconnects, power profiles missing). Countless times do I see a technician running driver updates on an unsupported windows build eventuating into more problems.
The main issues here are lack of education/time to become educated. As you said, its natural prioritization of tasks with no specific deadline. There is simply not enough engineers to sustain the amount of infrastructure we have to the best practice standard and everyone has their strengths and weaknesses.
In offensive security, its becoming a thing to check authors of vulnerable code to cross check it against other stuff they've worked on. Its targeting the inadequacies of engineers and developers based on what you expect to be in the "too hard basket".
This isn't a technical issue, its a management issue. There is X amount of work to complete, it takes Y amount of hours to complete to best practice, we need Z amount of people to complete this in a reasonable amount of time. What is a reasonable amount of time? it depends on who you ask and their level of risk acceptance given they've been educated in the manner.