Vendor we fired 2 years ago still has VPN access and admin rights to our backup system
Posted by SpecialistAd7913@reddit | sysadmin | View on Reddit | 68 comments
Started here three months ago. Been doing security cleanup and found VPN accounts for an MSP we stopped using in 2023. Contract ended, relationship over, but nobody disabled their technical access.
Five technicians from that MSP still have active VPN credentials. Checked what they can reach and it's bad. Domain admin on some servers. Full access to our Veeam backup environment. Read access to file shares with customer data. RDP to several production hosts. They could log in right now if they wanted to and we'd have no idea it wasn't one of our own admins because the accounts look legitimate in all the logs.
Asked around about offboarding process for vendors. There isn't one. When contracts end procurement closes the purchase order and that's it. Nobody tells IT to revoke technical access. We have a formal process for employee terminations but vendor relationships just fade away and their access stays forever. Started digging and found three other former vendors with active accounts. Consultants from projects that finished years ago. Implementation partners. A monitoring service we replaced.
The scary part is I only found these by manually going through account lists. No automated way to flag vendor accounts that outlived their contracts. No tie between procurement system and IAM. If I hadn't randomly decided to audit VPN access this month these accounts would still be sitting there. How do orgs actually track vendor technical access lifecycle when procurement and IT don't talk to each other?
zer04ll@reddit
This is a director or managers job and they didn't do it, vendor access should be managed by someone...
And folks claim requiring a password change every so often is bad, yeah for end users but vendors their shit could get hacked giving them access to ours, they are getting a new password requirement every 90 days at min. I do this to force vendors to at least be paying attention and if something breaks because they are not then thats also a sign they are not worth what you are paying.
Anonymous1Ninja@reddit
And? Did you fix it?
kurizma@reddit
no, because this is AI slop trying to farm info.
Euler007@reddit
My take too. Write the Reddit post after fixing the problem, not before.
Far_Gift6173@reddit
Sometimes it takes time. Perhaps he is at home at the moment
Euler007@reddit
Maybe. Anyways I should get back to work there's some smoke coming out of the server room. Just a quick coffee first.
Far_Gift6173@reddit
If the smoke has been there for years, the it's probably fine the next few minuts and you can also eat something
Euler007@reddit
I usually goes away on its own!
Far_Gift6173@reddit
OP already said, that he disabled the accounts.
THe issue he has is, that there is no proper offboarding process
Anonymous1Ninja@reddit
See i would've just setup a scheduled task to check for stale accounts and disable them on the DC.
No vpn without network level access
Far_Gift6173@reddit
Lots of options depening on the infrastructure
Ol_JanxSpirit@reddit
Might be a "better to ask forgiveness than permission" situation.
OneSeaworthiness7768@reddit
That’s the whole point of this post. Market research for their SaaS tool they’re likely working on.
Frothyleet@reddit
It sounds like he's a little lost on how to fix the problem and came here for advice. I think this is a legitimate use of the subreddit. Anybody here with mature policies thinks the solution is fairly obvious but if OP is newer to the game it's understandable for him.
Although for many of these posts I always wonder - OK, what did your boss say when you pointed out a gaping security/policy hole?
cvc75@reddit
Right, fix the immediate problem by disabling these accounts, and after that I don't see anything wrong with asking other people. Even if you go to your boss and point this out, it can't hurt if you already have some ideas to present on which processes you could implement in your org.
Chuck_II@reddit
Reading is hard.
"...these accounts would still be sitting there."
Anonymous1Ninja@reddit
If only there were some kind of language we could maybe program this in, and perhaps run it on something that controls these accounts, maybe run it at regularly scheduled intervals. IDK i would have to think about it for a little bit.
OneSeaworthiness7768@reddit
Don’t worry, they’re building a tool for that. 🙄
420GB@reddit
Curious how that works. Many servers not joined to the domain?
ethnicman1971@reddit
I get this for non-IT contracts but wasn't it IT that managed the relationship with the MSP? So, wouldn't IT know the relationship has ended? They would have likely even been the ones to end the relationship. Why wouldn't they be the ones to know that they had to disable their access?
TinderSubThrowAway@reddit
stop making sense.
ethnicman1971@reddit
sorry. I will go get my torch and pitchfork and join the outrage.
bitslammer@reddit
This is easily solved by having regualr access reviews and by having that off boarding process you're missing.
dracotrapnet@reddit
Back up those access reviews with account expirations. All vendors expire 3rd week of Jan, reminder to review 1st week of Jan. That gives you a week to assign the task to someone and someone to prod stakeholders that work with the vendors and get a response by the 3rd week. No response by the 3rd week and they are auto disabled and someone gets a phone call sooner or later "Hey my access is gone" if nobody responds.
WideAwakeNotSleeping@reddit
Exactly what I wanted to say. All our non-employee human accounts (vendors, contractors, etc) have set expiration date on their accounts (max 6 months). If the access is not extended, account just gets disabled and cannot be logged into.
marklein@reddit
We do similar, but more of a scream test. All outside account expire after 6 months, and we just leave them expired. They'll call if they're actually using it. So many just never call, indicating to us that they weren't using it anyway.
Entegy@reddit
Account expirations are an Active Directory thing though. There's no equivalent in Entra ID, so cloud-only orgs really have to have their policies in order.
bitslammer@reddit
That's pretty much what we do. It's all automated and if the account "owner' doesn't attest to the need for continued access they get deactivated and then removed.
Arudinne@reddit
We do this annually with an external auditor. It's tedious and time consuming, but it's also worth the effort.
bitslammer@reddit
Our is thankfully automated.
Arudinne@reddit
I have automated much of what get's requested with PowerShell scripts.
Sadly it's hard to automate responding to someone's questions in a team meeting.
bitslammer@reddit
Ouch. Ours is done view a combo of ServiceNow and Dell One Identity via email workflows.
bippy_b@reddit
At my old job we had to do this on a quarterly basis for AD, applications and database to pass client audits.
theoverseerer@reddit
We put expiry dates, 6 months for all contract/vendor accounts, along with email reminders to the initial requestor, 2 months, 1 month before expiry, to request extension.
RikiWardOG@reddit
We deal with the same issue with contractors, other depts never communicate end dates or that contracts are done etc We're working on getting some automation in place with our new HRIS, so we can then point fingers at HR lol
1z1z2x2x3c3c4v4v@reddit
Every vendor/external account gets a sponsor and expiration date. No excuses on this one. Accounts that haven't been used in 90 days also automatically get disabled.
Also, no one should have direct Domain Admin access without using some type of Privileged Account Management.
Multiple accounts for Admin, Cloud, and User level access. No one account (that is used) has access to everything. The more risky the system access, the more secure and locked down the account and process is to get authenticated and authorized.
HogginTheFeedz@reddit
These AI-generated spam posts are getting out of hand.
foxfire1112@reddit
Every one of them is themed as "im horrible at my job, let me tell you"
OneSeaworthiness7768@reddit
Because they want people to reply with their own issues so OP can try to solve them with their own vibecoded service. All the vibecoding SaaS guys are telling people “find a real problem people are having and try to solve it. Connect with people on Reddit to find pain points.” That’s what all of these posts are.
foxfire1112@reddit
Gross
CountGeoffrey@reddit
on a positive note, without a paid contract (even with one), it's super unlikely they ever actually used that access
OneSeaworthiness7768@reddit
Every market research post ends with this question.
JuniorCombination774@reddit
VPN access might not be your best bet. There's vendor access tools like PAM that grant temporary (only when you approve) domain/application/server access to vendors and removes all access when they're off-boarded. Permanent, untracked access is a nightmare.
desmond_koh@reddit
So, they fired their MSP and their infrastructure went unmanaged and improperly maintained resulting in gaping security holes.
Seems almost a little too cliche, doesn't it? As an MSP this is the kind of invisible value we deliver that budget-conscious controllers never realize when all they're doing is looking at the bottom line.
So, it's on you to develop one.
Rio__Grande@reddit
The truth is that vendor probably wouldn't even notify you when an employee left
zqpmx@reddit
We had VPN password and certificate expiration enabled.
But the proper way is to have a process to to deal with it.
paleologus@reddit
Setting expiration on AD accounts is what we do with any contracted users.
otacon967@reddit
Not acceptable. Refer this to legal and get them to bring contract/procurement team to heel. Their negligence is creating legal and infosec risk. Ideally off boarding workflow needs to be automated with at least 1 human approval. At a minimum it can be an email for smaller orgs.
ajf8729@reddit
“Domain admin to some servers”
That’s not how that works. Either you are a domain admin and have, gasps, admin rights to the domain, or you have local admin rights to a server.
phouchg0@reddit
We had accounts expire exactly when the contract did. Of course occasionally the contract would be renewed, the accounts would expire anyway, vendors would have to go home user til it was fixed
Anthropic_Principles@reddit
If it's any support, you're not alone in finding this. I found this and (much) worse in my last organization.
As others have already described the solution is regular reviews and time limited service entitlements.
DJDoubleDave@reddit
We do quarterly access reviews to flag stuff like this. They only take maybe an hour, I spit out reports of every account with admin access, or access to sensitive stuff, and go over them together in a meeting with the team. We add subtasks for anything that needs followup, and send to managers for approval. By having it all in a ticket it makes audits easy (I'm at an org that's heavily audited).
In the AD side, we do use expiration dates, and the account sponsor needs to request they be extended.
Still it's not perfect, we fight with a similar gap. In a perfect world the account would be disabled as soon as the contract ended, but in practice the IT side often doesn't know the real end dates, especially in a situation like above where a vendor gets fired. The sponsors just don't send in an onboarding ticket, and so the accounts persist until either they expire or get flagged in an access review.
sryan2k1@reddit
Every vendor/external account gets a sponsor and expiration date. At larger places I've worked there was some automated renewal workflows that would send reminder emails to the sponger/etc but most places the accounts just expire unless someone is actively telling them to stay working.
FrostFish88@reddit
Same process for my place. Vendor accounts expire every 6 months. Automated internal email send out to appropriate staff 2 weeks prior to expiration.
Affectionate-Cat-975@reddit
How was it unaware of the MSP offboarding?
punkwalrus@reddit
I wish this was uncommon.
We had a contractor with a 4-month support contract. He had gotten AWS root keys for one of our cloud accounts (well, we had him set it up). Literally keys to the kingdom with no expiration. AWS had to alert us, and it had been there for nearly 2 years. Nothing bad happened, but if someone gained access, they could have done anything to the account. Once we changed all that, we found the old root keys were showing up in Google searches with a ton of Github accounts. After some consideration, we justy closed that account since it wasn't being actively used anymore.
Anonymous1Ninja@reddit
I just want to point out the difference between system administrators in this sub.
- Some of us would just setup a scheduled task that identifies accounts that have not been active in X amount of days and disables them on the domain controller so that network level authentication is removed so this is never a problem
-Then the others would like to setup a process, that needs to be followed, by X Y and Z and is never communicated and never is followed by anyone other than IT, while going through multiple avenues to identify an account and where it has access and to what system.
Deprof_kan@reddit
That’s exactly why it’s important to have an overboard protocol and to carry out audits from time to time.
Because there is always a risk of getting distracted by very important tasks and forgetting to do something :)
It’s good that everything ended wellю
rootkode@reddit
That’s your own damn fault
BK_Rich@reddit
Use the account expire (end of) option in AD, set the dates for when the project is over or do maximum times like 1 month, so no permanent access and if the vendor still needs access, they will reach out.
Justneedsomehelps@reddit
I see it all the time. You need that off boarding process and review even your ms partner relationships as thats another hidden way they have access.
ohfucknotthisagain@reddit
You should maintain a folder with information for each vendor.
It should have everything you need to know to work with them, such as Its sales/licensing contacts, technical support info, purchase orders, contracts/fulfillment, authorized personnel, etc.
Include in that documentation a list of configuration changes (firewall, VPN, etc), internal accounts, and special access requirements. This should be updated whenever new versions or services are rolled out. When the relationship ends, nuke it all.
PanicAdmin@reddit
I'm a MSP, and credentials disabling is part of my onboarding process.
Almost any AD has some kind of accounts like that.
I also regularly use the account validity "scream test", it works.
0verstim@reddit
You came here to talk about it before cutting them off? Youre not doing your job...
OkEmployment4437@reddit
This is exactly why external access needs its own joiner/mover/leaver process.
Immediate fix: disable every inactive vendor VPN account and external privileged account now, then review logs on the VPN, backup platform, and AD/LDAP to see what actually got used. I would not wait for a formal review cycle when former vendors still have backup admin and server access.
For the durable fix, every vendor account should have four fields on day one: internal sponsor, business purpose, hard expiry date, and scoped access. If one of those is missing, the account should not exist. No standing admin rights unless there is a very good reason.
Then add a quarterly recertification: sponsor confirms the access still belongs, otherwise it expires. Keep it simple if you have to. Even a spreadsheet plus tickets is better than nothing.
The key control is tying procurement/legal events to IT actions. Contract ends, SOW closes, or vendor changes personnel should automatically create an access removal ticket.
If vendors need privileged access regularly, use brokered or time-limited access instead of permanent VPN accounts.
AppIdentityGuy@reddit
The sad tbing is that this not uncommon. I come across it all the time. I would suggest that getting rid of any account that has not logged in over a year would reduce the ADDS attack surface by at least 50% if not a lot more..
ABlankwindow@reddit
Quarterly user audit at the least. even in cases like mine where both sides work together shit still falls thru the cracks which is why we do quarterly audits. We also have a script that auto purges logins after no login for 90 days. When able we also set user auto expirations to contract dates
bageloid@reddit
...they haven't done an access/entitlement recertification review for 3 years?
woof