Vendor Installed NinjaRMM Without Consent Bypassing Security - What Would You Do?
Posted by Typical-Hornet-1561@reddit | sysadmin | View on Reddit | 87 comments
I was recently reviewing software on a server used for a vendor's product when I came across NinjaRMM in the control panel installed more recently than any of my logs had shown the vendor remoting into the network.
I know the vendor deploys code and product updates via Octopus Deploy (PowerShell Initiates a Network Connection to GitHub) as this had been flagged by the firewall previously and allowed since it was deemed relevant to the vendor's product.
I then found the logs showing all of the system & network information being sent back by the NinjaRMM agent and am quite surprised at the data that is leaving the environment that was set up without any sort of consent or notification to our IT team.
Is this normal behavior from a software vendor? Would you be concerned? How would you approach the situation?
BitProber512@reddit
Id definitly be contacting the vendor to confirm.
BitProber512@reddit
If the vendor is changing how they admin software on your hardware thats something that definitly should be communicated ahead of time. Major Sketch on that.
sryan2k1@reddit
The notifications may have been ignored or going to the wrong person like billing
Xelopheris@reddit
This. Many companies do not necessarily have a "technical guy" contact in their CMS. Any kind of update like that will potentially go out to a manager who signed up for the product years ago, and he isn't there anymore so his emails are going to a director or VP who isn't going to read them or know anything about them.
BitProber512@reddit
Meh im thinking more along the lines of a supplychain attack.
BitProber512@reddit
I must add. YOU LIKE ME, YOU REALLY LIKE ME!. Apparently.
Ok-Juggernaut-4698@reddit
This is unacceptable
GeneMoody-Action1@reddit
Hells to tha no!
Without an explicitly agreed on paper trail, kill it with fire.
And while it is burning, hold that vendor to the candle, who authorized, when, where, signed off on, etc...
bukkithedd@reddit
I would have so many words and none of them polite to any vendor that did that on our servers.
I'll gladly give the vendors the access they need when they're installing their shit, but I will NOT tolerate unattended access to our servers through software like that. Period, full stop, do not pass go, fuck right off and continue to fuck off until you've successfully fucked off to another galaxy.
If you have to access our shit, you'll give me a call and I'll hook you up. Every single time, no exceptions.
Kurgan_IT@reddit
Most software vendors pull these stunts not because they are malicious, but because they think it's useful to them and they just don't care / don't know anything about security.
Shares with everyone full control, chmod 777, remote management software like anydesk o teamviewer installed without consent, etc.
As a consultant I run into these issues more or less everywhere.
KingDaveRa@reddit
"We're going to install LogMeIn so we can give support if we need to"
No, no you are not.
Financial-Chemist360@reddit
Those are the same people who call and say "we need you to just open up the firewall".
lemachet@reddit
But radio silence when you ask them if it's inbound or outbound and what dst IP:port and what src IP:port
Financial-Chemist360@reddit
No, you've missed the point! They don't know a firewall from a particle collider. They just see the firewall as the problem that's keeping them from getting to their objective so they want it removed.
way__north@reddit
firewalls are known to cause trouble, best to just set any any accept
MedicatedLiver@reddit
The amount of vendor trash that "require" local use admin rights to even launch their software is astounding.
Like, I get it back in, say, 2013 when everyone was switching to Win7/8 and running old software, but bullshit on anything after 2009. You've known about UAC and how it works since Vista.
FFS, Win10 came out in 2014. Vendors have had MORE than a decade just on that. Almost 20 years now since UAC came out period. But some CLevel gonna get enough kickback to approve the shittiest software.
ShadowSlayer1441@reddit
Please run this debug command: sudo chmod 777 "/*" && setenforce "0"
kozak_@reddit
Grants full read, write, and execute permissions to all users for every file and directory under the root directory, making the system insecure.
Disables SELinux enforcement, removing security policies and leaving the system vulnerable.
Kurgan_IT@reddit
Actually breaks the system because a lot of software stops working if it detects wrong permissions on critical files.
ShadowSlayer1441@reddit
Yeah, this is more r/shittysysadmin. The setenforce is a genuine debugging option if you believe SELinux is causing the issue as it doesn't delete any policy only disable enforcement until reboot. If the issue persists after setenforce it's definitely not SELinux. Obviously you have to be careful if the computer has sensitive data and/or is connected to the internet, but I mean it's likely to be compromised in a few minutes. I would reboot immediately after confirming the if the issues persists. The chmod stuff was pure shitpost, an absolutely terrible idea, but I mean it could fix a number of issues.
If someone saw my comment labeled debug commands ran them without googling what chmod or setenforce did, well they were already r/shittysysadmin.
AlligatorFarts@reddit
Surely that'll debug... something.
PM_ME_YOUR_GREENERY@reddit
I have one better - RDS server, vendor requires users to be admins. Of the entire server. It's needed to be turned back on more than once.
ollytheninja@reddit
Agree. Is it normal? Yes. Should you be concerned? Also yes. How do you approach? Depends on your real and agreement with them and the nature of the data you’re processing. I’d just say security monitoring flagged it and you want to check if this is intentional. They’ll either say yes, in which case you need to figure out if it’s a problem for you, or they’ll say no and it’s a security incident.
chemcast9801@reddit
This is the answer and also my suggestion OP. Without details of what the vendor is providing and such that’s about the best advice you can get.
-MoC-@reddit
1st thing I would do is find out when and how they got it installed and how you were not aware of it. And make sure you have things in place to stop similar happening again or at least alerting you when it happens.
Assuming you still need the vendor, check contracts and make sure you didn't agree to it then contact the vendor and tell then there is a breach in your security policy find out what they are using it for and come up with a solution you control to do the same thing. then discuss service credits once its fixed.
if you don't want them use it as an excuse to get out of contracts without paying... assuming its not agreed to in the contract.
r-NBK@reddit
No way that would fly in my org. Servers do not have Internet access other than extremely limited and vetted domains - and we would never allow a server to have access to GitHub.
If a vendor of ours was caught installing a remote connection or management tool. Our CISO and our CIO would be contacting them very quickly.
Secret_Account07@reddit
Would need a lot of context here.
On its face- yeah that’s a security concern.
What is the agreement/contract with this vendor. If remote access is included for them- was a VPN and RDP supposed to be used? Curious how/if they connected remotely prior to this install.
I think if it was me I would follow security response. Go through logs see which user installed. You could reach out to that contractor for explanation or just disable the account until you verify it’s not a compromised account. I guess isolating the server is an option but it sounds like this may not be needed.
I agree that they probably overstepped and created another vulnerability on what sounds like a public facing server? Or perhaps firewall allows this connection without VPN? Not sure.
But I try to give people a little slack for mistakes that don’t have major consequences. Likely did it with good intentions but needs to understand this cannot happen. Likely if your org raises hell over it this person could get fired. If they do good work as a vendor, I’d try to resolve this behavior with their mgmt.
I remember in my old days of doing desktop this was an acceptable practice for hard to reach computers. Security and times have changed though.
FWIW this was a good demonstration of locating a vulnerability/issue, investigating logs, locating offending account and remediating. Now your mgmt just needs to decide if they want to go to war over it.
Papfox@reddit
If one of our vendors did this, our IT Security and Risk Management department would do their nuts. The endpoint security management bot on that server would shoot the thing in the head and possibly trigger the machine being quarantined on the network. There would be meetings with the potential to spoil our days.
The vendor being shown the door permanently would be a definite possibility
BeyondRAM@reddit
Uninstall it, kinda pain in the ass sometimes with these RMM agents
JustSomeGuy556@reddit
After a few problems, we don't allow vendors to install any remote software on servers of ours at all. All vendor activities must be done via screenshare and with one of our sysadmins supervising.
Vendors do not like that.
We don't care.
And our CIO has our back on this. It goes into all of our contracts.
Pisses off the vendors sometimes, but my give a shit meter is busted.
Upbeat-Carrot455@reddit
Yup. We piss Avaya off since they can’t just reach in and check on license usage. We frustrate all people who help. But it doesn’t matter that’s the security posture if they want to do business with us. Screen share or onsite visits.
architectofinsanity@reddit
I work for a company that offers both options for support. It’s time others accept it and move forward.
All it takes is one bad experience from one vendor and our option of demoting it is gone.
simonjakeevan@reddit
This is the way.
IllustriousRaccoon25@reddit
Or get something like BeyondTrust Privileged Access to only let them in when you approve, then record everything they do.
ilbicelli@reddit
We do somethin similar with Apache Guacamole: every vendor has an access to our gateway and sessions are recorded.
joefleisch@reddit
Is the server vendor controlled?
If not, start an incident response.
I would want a server segmented and not on domain if a 3rd party used their NinjaRMM unless we had a contract for the usage and knew about it and could audit it.
dorflGhoat@reddit
Agreed. Escalate and treat as a potential breach until someone can confirm it’s authorised to be there.
iiThecollector@reddit
I agree with you on this
andytagonist@reddit
You let a vendor bypass your security?
cybersplice@reddit
I'm working on getting Ninja into a client as the first candidate for our migration from Automate.
We have exchanged a zillion emails and a change control for consent.
This is not OK.
The ninja agent gives root/system level access to any machine it's installed on, including sensitive machines like DCs.
They could just launch PowerShell and add/remove users on your domain.
I assume they have this capability anyway since they were able to deploy it.
throwaway0000012132@reddit
Why your servers have full access to the internet?
Anyway, contact your vendor to clarify this issue and reassure them that they can be processed if a major incident caused by a security breach on their part happens, due to financial and reputation loss. If it's on Europe, fines are also astronomical.
It's 2025, everyone should be responsible for good security practices
boukej@reddit
That's why I restrict Internet connectivity on servers, and only allow what is really needed, and that's not much.
MBILC@reddit
I wish more people did this, Servers should be default blocked from internet access period, they do not need it, unless it is hosting something which even then, should be run through proxies via a perimeter device to control access.
boukej@reddit
Yes. I prefer a mix of a VM running Squid proxy with a white list and where required some outbound firewall rules which allow the bare minimum.
This is to ensure updates and our RMM-client to connect to 'just there' where it should.
no_regerts_bob@reddit
definitely not normal. are you sure the vendor installed it? i would want some answers
DSMRick@reddit
I would say normal, but still not acceptable. Vendors do all kinds of shit like this.
jmbpiano@reddit
I would say vendors pulling stunts like that is normal, but the discrepancy in the logs is not.
VTRnd@reddit
If you have a support contract with the vendor its probably normal. NinjaRMM is used to monitor applications and perform updates. If there are issues, NinjaRMM can solve things automatically.
So I would say depending of the requirements you have to the vendor its normal. If you dont have a support contract it isnt normal.
AMoreExcitingName@reddit
Ninja automatically updates utself. So if the install date seems very r3cent, it's probably that
free2game@reddit
Uninstall it using the uninstaller in it's program file directory. It doesn't clean up the registry so new installs will error our if they try to reinstall it.
skywatcher2022@reddit
He's outta here, don't pass go, don't collect $200. However do confirm it was installed by him/them with there login first. Then determine the extent of the damage and send the bill to the company that dispatched him to your site. If that includes reinstalling all the machines on overtime for 10 people so be it. We dont all vendors to install anything on any server at any time for any reason. It must go through our security evaluation and our it staff must install it in a jail and proven well before installation on our network. We generally don't even allow vendors internet access without being in an isolated network segment or they need to BYOI cellular/starlink etc.
allllusernamestaken@reddit
i think you need to rethink your entire setup. You're cooked bro.
JMejia5429@reddit
blacklist the app / domain and let the vendor reach out to you that they can't connect and then you can start the convo re unauthorized software that they installed and how they are liable plus you want compensation for putting your company at risk (obviously involve legal)
AppropriateSpell5405@reddit
This is their hardware on-prem? This is a server you gave them access to? This server has shared workload? Also, is this an actual physical server, or you spun up a VM and gave them access to that?
Ultimately, the server should really have network isolation and only access to what it needs. In theory, if you've set things up properly on your end, you could give them full control within their own little world without having to worry much about what they're doing.
malikto44@reddit
I'd treat this as a security breach and get the CISO and legal. I'd treat NinjaOne as an install of a RAT on a secured network.
A70M1C@reddit
Coming towards the ass end of a multi year refurb of huge entertainment complex. I am the perm operation IT manager. 1001 contractors on project amd they keep Dropping the portable team viewer on management servers. Got rid and formally raised with project head 8 times in a month, they kept on ignoring me.
So everytime the vulnerability scanner found the fucking thing I Disabled the account for every staff member of that company until they completed a remote access review and retraining on the 2FA VPN.
After third remote access review I never found team viewer on the network again.
sudonem@reddit
Speak to legal, look at the contracts and find an exit strategy. Whoever this vendor is, they made it clear that security isn’t a priority (or even a baseline skill set) of theirs and they cannot be trusted.
enbenlen@reddit
Usually cases of gross incompetence like this are grounds for contract termination with every MSP I have encountered, so it shouldn’t be terribly difficult.
Itsnotvd@reddit
I could get fired if I ignored something like this.
1st step talk to manager. Emails to follow to cover myself and may be cc'd to others depending upon manager response.
No way on earth where I work would some remote access be allowed on a server by a vendor and admin access. If this was done and not authorized, that vendor would be history.
crnkymvmt@reddit
Remove it now, revert the likely new firewall rules they put in place, start the scaffolding on a vendor support policy based on what youve learned here. Really gives you peace of mind when the rules of engagement are clear to everyone.
afiendish1@reddit
It definitely should have notified prior to push, but accounting would have ignored the email if it went to them. Seems like they already had remote access and control. I would probably light them up, but most of the people collecting my budget are not interested in putting our relationship at risk.
hops_on_hops@reddit
Security incident. Even if you know who probably did it, you've had a security breach.
Initiate your incident response plan. Shot down the system Call in whoever you need to call in. Call the vendor. Review their contracts, etc. Then probably only allow them supervised access in the future.
RelativeID@reddit
Punch them in the face. No seriously, I would complain to my manager.
simonjakeevan@reddit
And then punch your manager in the face
BoltActionRifleman@reddit
A few years ago we implemented a requirement for all vendors to not be allowed to connect to our systems without having an employee approve their MFA. Many people barked at first, because it’s just “so much easier to give them unfettered access”, but they all eventually fell in line and now no one seems to mind. I’d highly recommend this method to anyone wanting to keep their vendors under control.
keitheii@reddit
If it were me, that vendor would be replaced immediately. I take security very seriously and there is no second chance when it comes to unethical behavior from a vendor, and I won't care what excuse they give.
nichetcher@reddit
Approach him directly and ask why he installed it. Then come back and we will judge whether he should be lynched.
nichetcher@reddit
If somehow your vendor was able to just “install” NinjaRMM, then you gave him admin access that he should not have had unless he’s allowed to install whatever he wants.
MountainDadwBeard@reddit
I mean if you're giving powershell access(?) your pants are down anyways.
Your vendor management system and SLA should always clarify remote access privileges.
Since most people aren't clarifying this I'm personally seeing quite a few vendors with undeclared backdoor access.
gabber2694@reddit
This would be contract ending in most of my environments.
Show me the change control Show me where you state this requirement Show me the notes from our sign off Show me how you secured this installation
This is serious cowboy activity and will certainly not be the only bad decision they make.
6Saint6Cyber6@reddit
Check with your team/boss to find out if it was authorized. If not, then it is an incident - follow your data exfiltration incident plan.
If I found this in our environment? I would nuke access to that server before I did anything else, but I'm a block now, check later kind of person.
port25@reddit
Did we just become best friends?
Helpjuice@reddit
First thing I would do is look at the contract between the company and vendor and get it fixed so all work is done by the company with vendor guidance if applicable. Never let a 3rd party run amok on anything you have responsibility for securing without assurances and processes in place so all changes are reviewed and controlled. This way if a security issue is introduced into the environment it is much easier to target when, where, how, what, etc.
NCC1701-Enterprise@reddit
I would be willing to bet in the fine print of your contract with them they are allowed to install software for remote management and monitoring.
ryanlaghost@reddit
Yup, sounds like some fine line nonsense lol
port25@reddit
That's just one thing you found... For us that alone would be a P2 incident. Official protocol is the appropriate response. Are you the boss? I wouldn't want to be in the middle here, unless you signed that contract you have no dog in the hunt. Clean quarantine report. Everyone here has excellent feedback as well. Good idea asking reddit, this sub is surprisingly insightful. (After we draw blood)
ISeeDeadPackets@reddit
Not sure what your position is there, but in my environment my first response would be to completely disable their access and reach out to my account rep for an explanation. Assuming it really does exceed their authorization it could be grounds to terminate the relationship. Ninja's a solid tool but that doesn't mean it's OK to install it without permission. In fact Ninja themselves would probably not be happy to learn they were doing that.
macr6@reddit
Key phrase here is if they don’t have authorization to do this and if you don’t know OP, you could get into trouble. Make sure. But bring it up immediately.
JediMind1209@reddit
Who gave them access to install software?
iwinsallthethings@reddit
Honestly, I'd let me boss know. Then probably follow up with security.
My boss would immediately ask for access to be revoked. Security would get all in a huff and start looking at things. We would wonder why security didn't catch this will their 347 tools on each end point. My boss would then ask me to get legal involved for contract purposes. I would then be done with the process until i'm told to either grant access, remove the app, or delete the VM.
LonelyWizardDead@reddit
whats the vender do for you? is one question.
i'd say no its not normal, and shouldnt be allowed unless agreed and reviewd.
what else are htey deploying and were, what information are they collectiong and why?
jOvAfEiA@reddit
I migrated our entire fleet to Intune over new year, getting rid of so much legacy bullshit. 1st day into everyone working with the new system, a software vendor installed fkn AnyDesk on the bookkeepers machine.
--> Currently planning the migration away from their product :)
Bubby_Mang@reddit
You're talking about a pretty standard block the port and e-mail the guy maneuver bud.
serverhorror@reddit
Hand it over to legal and sourcing so they handle it.
Smaller company? Give the other side, the boss directly, a call and a pep talk.
Brad_from_Wisconsin@reddit
Are you sure of where the data is going?
If the install does not align with the vendor access, I would spend some time looking at other ways the software ended up on the server.
I would still blame the vendor on general principals.
--We had a software vendor, the business unit went with cheapest vendor, that was a one man shop and he would sign in at night to install updates or make modifications on the system. We pushed the server to an island but we would still get random 5 am calls blaming us for service outages on the system.
Icy-Ice2362@reddit
We're trying to move away from Data as Value to Data as Liability.
Nothing like a little National Security Act = Life in Prison to make a person think twice.
sryan2k1@reddit
Talk to your AE. It's possible this is in the T&C's or someone in your organization ignored notification of this.