Am I overreacting? MSP using shared global admin, no pim, admin account = standard account

[-]

Master-IT-All@reddit

Not over-reacting. That is sadly pretty typical.

[-]

RestartRebootRetire@reddit

I was hired as IT Manager into a SMB using an MSP that had 300+ clients.

When I got here, all Windows firewalls were off, all file shares were "everyone, full control" and we even had a utility PC with RDP exposed to the Internet.

Their Tier 1 grunts would do stuff like this and wreck things over time, then their $300/hr engineers would be called in to fix what those guys broke, with hefty project fees.

[-]

I've gotten a bad taste for MSPs since then. The owner of that business goes to annual conferences in our sector and sells himself to SMBs and makes outlandish claims about tech he was first to do 30 years ago.

I'm sure some MSPs are probably good if all your company does is use a web browser and print once and a while.

[-]

Salamandro@reddit

"everyone, full control" on file shares is best practice. Unless we're talking NTFS permissions, then no :-)

[-]

RestartRebootRetire@reddit

Yeah, it was definitely NTFS....I actually watched one of them try it again during a live support call.

[-]

Candid_Candle_905@reddit

Nope, thats a legit red flag - shared GA with no PIM is straight-up reckless, and Microsoft’s own guidance is to keep GA count tiny and use PIM for privileged roles. https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/best-practices
Separate admin accounts, just-in-time activation, MFA &break-glass accounts.... the current setup is basically “please audit us later”

[-]

thvnderfvck@reddit

Microsoft’s own guidance is to keep GA count tiny and use PIM for privileged roles.

Microsoft's guidance is to spend more for premium security features. Whodathunkit?

[-]

cosmic_orca@reddit

This is fine for the MSP itself, but companies that MSPs support are often smaller companies that refuse to pay for Entra P2 licences, which PIM requires. But yeh no reason not to have separate admin accounts.

[-]

Salamandro@reddit

I mean, with one shared GA account you do adhere to a tiny count :x

[-]

bakonpie@reddit

most MSPs are dumpster fires example #7496

[-]

ItaJohnson@reddit

That is my take on them too. Especially the small ones. I worked for one that employed what I feel are questionable practices. Thinking 8.8.8.8 qualifies as a redundant DNS server being one such practice.

[-]

Teguri@reddit

credentials in a notepad on shared VMs with client....

[-]

BreathDeeply101@reddit

I'm going to agree with /u/SpiritactDevOps and say most companies period.

I worked for a MSP for almost 20 years. We weren't perfect and we took over from a fair number of other MSPs were we found settings and practices that were scary..... but then we consulted with an awful lot of companies with full-time IT staff and saw the same thing. When I left the MSP and landed a new job I treated the new place the same as onboarding a client at the MSP and did a fairly full audit as part of my onboarding and found a number of issues to bring up.

Every new employee is a set of new eyes and experiences that can help make a better environment.

And yes, they can also be the chaos monkey and make things worse.

[-]

Spiritact@reddit

At this point I think nearly every company is a dumpster fire or has one somewhere.

[-]

demalo@reddit

We keep it out back with the other dumpsters.

[-]

Afraid_Baseball_3962@reddit

Only one dumpster fire?

[-]

demalo@reddit

The others are probably on fire too. They have department names: HR, Finance, Management… not my smoke, not my dumpster fire.

[-]

timbotheny26@reddit

It's just one big pile of rusty dumpsters haphazardly stacked in a big pile and set ablaze.

[-]

timbotheny26@reddit

Outsourced overseas MSP

Yeah, that explains it.

[-]

m4tic@reddit

Raise the flag on every concern, create a paper trail, and ask for clarification on items that are denied or don't get traction. Someone already posted the CYA wiki.

[-]

Nu-Hir@reddit

Who answers to who? Does the MSP answer to you, or do you answer to the MSP? If changes need to be made, do you (or someone above you) have the final say, or does the MSP?

If they answer to you (or your boss) then it sounds like you need to start pushing new policies that need to be followed. If they make final decisions, it sounds like you're going to have an uphill battle.

[-]

quantumhardline@reddit

Yes thats bad. See what their contracted to handle, depending in your role, figure out what framework your following or like CIS IG2/IG3 then see what controls they have in place. The MSP may be using something like CIPP etc. Depending on size of org some of these are now laws in certain states etc to avoid punitive damages. Also if ya’ll dont have a large cyber policy, get a stand alone and that will also have certain requirements and they can build you a risk in $ like we had one showing $5-$30M in risks via 3rd party and that will get you CFO etc support for changes.

[-]

PigeonRipper@reddit

Yes this is obviously bad. Next question please

[-]

Blue_Kayak@reddit

Red or Blue? And a follow-up if you’ll allow it: up or down?

[-]

PigeonRipper@reddit

Blue, like the colour of my favourite kayak :3

[-]

Kroan@reddit

What's the colour of your least favourite kayak?

[-]

DaCozPuddingPop@reddit (OP)

lol I just needed to share - I'm the sole internal IT guy for the moment and I am moderately shitting myself.

[-]

AnonymooseRedditor@reddit

Prepare your documentation and 3 envelopes. Do you at least require MFA

[-]

PatchUrStuffz@reddit

Yeah this would be priority #1 for me: Document the current state of everything, build policy for best practice & acceptable use with least privilege, get leadership buy in, start implementing change.

[-]

GardenWeasel67@reddit

You aren't just IT. You are the fall guy. Run.

[-]

alphaxion@reddit

If you're the company IT dept, can't you set up individual standard and -admin accounts and tell them that corporate security policy requires this going forwards for regulatory and standards reasons? If they fight you and demand to know which ones, you should be having a convo with your company legal dept about how to cancel their contract and either hire in-house going forwards or seek a better MSP (good luck on that front).

Get yourself a SIEM solution set up (elastic for example) and ship your auth logs to it. Build dashboards that focus visualisations on auth and audit that they're actually using the accounts you gave them appropriately.

Document that you have requested they follow new procedure.

If your MSP is the source of a compromise, you have covered your back, especially if you have documented times you have discovered them not following procedure and reached out to them to correct their behaviour.

[-]

PigeonRipper@reddit

Eh these situations keep us employed.

Just read this every morning https://en.wikipedia.org/wiki/Cover_your_ass

[-]

larsvdmeyde@reddit

Question. If you want MSP to have personal global admin accounts, you will have like 20 global admins. That doesnt seem good either right?

[-]

FartInTheLocker@reddit

Average MSP vibe lol, at my last job at an MSP we used to all have access to root windows domain admins, from senior engineers to apprentices, used to reset AD passwords lmao

Some wild shit happens at MSPs man

[-]

TabascohFiascoh@reddit

100%

I started my life in one. Looking back it was the fucking wild west. Hugely incompetent.

[-]

assassinboy4@reddit

I'm in a place like this right now, new tech got let go a while back after rebooting a critical file server in the middle of a day. Full access, all of the places, all of the time.

[-]

TabascohFiascoh@reddit

I have stories for days man. for DAYS.

[-]

1z1z2x2x3c3c4v4v@reddit

You work to get skills and experience. Get yours then move up or out as quickly as you can. This place sounds like a dumpster fire.

That said, it also sounds like you will have full rein to learn as much as you possibly can, while also being the fireman of the org.

Use your time wisely.

Once you realize you are no longer learning new in-demand skills, and are just putting out fires all week, thats when you move on.

[-]

DaCozPuddingPop@reddit (OP)

Positive side, it's going to be real REAL easy to show progress being made and collect a fat bonus.

Long as they don't get fucking compromised in the interim.

[-]

1z1z2x2x3c3c4v4v@reddit

Carpe Diem!

[-]

lanky_aesthetics@reddit

you're walking into a compliance nightmare, especially in finance - shared accounts and always-on global admin will tank any audit. First thing monday morning document everything you found, then get your manager and security team in a room with the MSP to explain why this setup needs to change like yesterday

[-]

beren0073@reddit

What does your company policy say about privileged access management?

[-]

DaCozPuddingPop@reddit (OP)

I'll let you know as soon as I am done writing it.

This is so bizarre - company is 6 years old and operating like a fucking startup.

[-]

TheBestHawksFan@reddit

A six year old company is not that mature and is perfectly in range for a startup. If you're their first in house IT person, none of this is surprising.

[-]

GardenWeasel67@reddit

Nope

gif

[-]

aguynamedbrand@reddit

It sounds like the MSO is not being managed by your IT department.

[-]

DaCozPuddingPop@reddit (OP)

Yeah, I think that's largely why they decided it was time to bring in a dedicated IT resource. They don't know what to tell these folks to do - and to your point, they're not gonna be proactive.

The list is growing rapidly. Oy vey.

[-]

BreathDeeply101@reddit

Any new position is ripe with improvements that can/should be made. Willing to bet you will find more scary things, if you haven't already. I worked at an MSP for almost 20 years and every new client we took on had a fair amount of 😐 and 😬 set up by either other MSPs or on-site IT. Build a list, prioritize, bring up the things you need buy off on versus just being able to implement by your own authority.

[-]

ItaJohnson@reddit

Or for other reasons. In my experience, they try to do as little as possible. Work being done for one client that is working that isn’t being done for a potentially more lucrative client.

[-]

FlameBeast123@reddit

Finance industry with no PIM and a shared global admin? Thats not just bad practice, thats the kind of thing that gets flagged in regulatory audits. If you have any authority at all, getting that MSP's access scoped down should be priority one.

[-]

serialband@reddit

MSP is incompetent, or they've lied about their abilities.

[-]

UserProv_Minotaur@reddit

This is several major red flags

[-]

According_Square2742@reddit

I’d be job hunting. Like yesterday.

[-]

TheBestHawksFan@reddit

because you have to do work? CYA and this is pretty easy to resolve.

[-]

TheGenericUser0815@reddit

Oh, wow, I work for a real small company (25ppl) and een we are better at account handling.

[-]

Nakenochny@reddit

If it’s banking, you’re in for a world of hurt when you’re examined by the Fed. If it’s accounting, that seems par for the course.

Source: worked in both. Banking is about 10x as regulated as accounting.

[-]

DaCozPuddingPop@reddit (OP)

It's a weird company...venture capital meets biotech...

So lots of oversight incoming for sure. Time to get a sec assessment done STAT.

[-]

Nakenochny@reddit

Ahhhh, so no real regulation at all. That’s a rough spot to be in, especially if they’ve never been breached before, because they don’t know how badly things could go wrong.

For me I’d at minimum set the MSP account to needing to be activated. If I was feeling a little more vindictive I’d remove GA and give them all the roles that would cumulatively make up GA.

Recommend CIS benchmarks for trying to up your security posture and mitigate some of the (almost certainly) gaping holes in your tenant. If you’ve got a P2 license you can put some good CA policies in place as well. If you do that though, make sure you’ve got a break glass account in place that’s always excluded from CA policies, preferably with a FIDO key for MFA.

If management won’t let you do things, be sure it’s all in writing so when the shit hits the fan you’ve already CYA’d.

[-]

fraghead5@reddit

the always on GA for a small set of admins is fine in our world, we are 300 employees and 2 have GA always assigned, the other folks that need any admin stuff use PIM.

Shared accounts are very much a no-no for everything except a few random tax portals that finance has to use, we have a company 1password account and have shared finance vaults where the 3 or 4 shared logins are kept and passwords rotated.

[-]

ExceptionEX@reddit

I mean, what are you paying for, do you have the p2 for PIM?

I don't love any of that either, but there isn't enough details here to know why.

[-]

DaCozPuddingPop@reddit (OP)

They don't have the P2 - but they're gonna real soon. Holy crud.

[-]

XL426@reddit

I know of an MSP that just a few years ago was scared as hell of enabling MFA on their (shared) admin account for each client in case they got locked out.....they walk amongst us

[-]

AppointmentIll9358@reddit

It’s probably being ran by non IT leadership

[-]

AppointmentIll9358@reddit

At my MSP we had global admin 😂, what a shit show

Just so your job as best as you can. If you don’t own the deployments then don’t make it your problem.

[-]

rickAUS@reddit

When I worked in MSP world:

1\ GA was restricted to only the highest techs, everyone else relied on whatever we had access to via GDAP, JIT/PIM
2\ Yeah, this is basic stupidity. Make your own admin login and move the role
3\ That's just stupid and defeats the purpose

You might be the only IT person but the MSP works for you. If you aren't happy with their security or means of access, change the requirements on how they get it. Probably 90% of their work can be done with global reader, user admin, exchange admin, sharepoint admin and teams admin. At least, that was the experience I had with tenants I needed to JIT roles to do stuff. Absolutely zero reason to be using GA as their daily driver in your environment - least access is best practice.

[-]

Salamandro@reddit

This was us, a year ago. I'm very glad I've introduced all these concepts into how we access customer's tenants by introducing CIPP and GDAP usage, accessed by secondary admin accounts with phishing resistant MFA auth. Internally, everything is done by PIM.

[-]

desmond_koh@reddit

Presumably your company hired an Overseas MSP because it was cheaper. Well, sorry, but you get what you pay for.

If you see proper IT management as optional, then this is what you get.

[-]

Entegy@reddit

There's a lot wrong here of course. GDAP doesn't cover everything, so shared MSP GA login can be normal.

For us, the MSP has such a shared account but the credentials are stored in a password manager with an audit log. We occasionally ask for an export of the log so we can see who used the credentials.

For yourself, GA on your normal account is fine if it was backed by PIM, phish-resistant MFA, and email notifications when activated.

You have some changes to fight for! And don't forget to CYA.

[-]

OkAssistance7072@reddit

Yea... thats fairly normal.

Now that you're global admin, start making things better. Come up with a plan and propose it in your team meeting or to your boss and let them know why the MSP is fucking horrible.

[-]

r1kupanda@reddit

Yeah that's an immature MSP for sure. They should be using GDAP with limited scope on the gdap relationship. If your licensing includes PIM, they should also configure that. If not, bare minimum of a separsted account with strict controls.

There are several compliance frameworks that the shared admin account would fail... I think you need to have a conversation with them to see if they can actually meet your businesses' needs.

[-]

MIGreene85@reddit

Honestly, the number of organizations that are setup poorly or don't adhere to best practices is far greater than the number of organizations that do IT well. Unfortunately, in many cases businesses don't know or care about the inherent risk created, either because they don't understand it or they just don't care. They are focused on dollars and cents, and you are talking about possible scenarios that haven't burned them yet. IT maturity and governance typically come as businesses increase in size and the stakes become greater, and/or they have already paid the price and been forced to invest more heavily to fix these issues after the fact.

[-]

Blue_Kayak@reddit

Brutal. I’m sorry you’re dealing with this. So your best to make it better and be sure to carefully document all of your progress (or lack thereof) 😊

[-]

crazyarky300@reddit

I agree with you. That’s not good!!