A government org recently audited their 4,000 device fleet. They found 4,000 more.
Posted by LizFromHexnode@reddit | sysadmin | View on Reddit | 76 comments
Kyle Manilal from Sizwe IT Group was doing a guest session for us at Hexnode recently, and he dropped a stat about a public sector audit that has been stuck in my head ever since. So this government dept kicked off an inventory audit fully expecting to find a fleet of around 4,000 endpoints. By the time the audit finished, they had logged 8,000. They were completely blind to half of their actual hardware!
I feel like a 5-10% inventory drift is just par for the course when dealing with large fleets (still not right), but missing half your endpoints is wild. It really makes you wonder how much of the global attack surface is just forgotten hardware sitting in a drawer somewhere.
Bernie_Dharma@reddit
Did some work for a global biotech firm a few years ago. They grew by acquisition, like many firms their size and had just been acquired by private equity who wanted to clear out the bad management, upgrade the business processes and terrible IT infrastructure. So we traveled site to site, assessing the current IT assets and upgrading them as needed.
We had just finished a 2 week stint in a foreign country upgrading three plants and were packing up to leave when the local GM asked us if we were also going to upgrade the other six sites. We asked for details and locations and called the CIO back in the US. No one at this company even knew those sites existed and they had to place calls back to the private equity firm to find out if they actually owned them. Six entire manufacturing plants that no one knew existed. We flew back to the IS while the lawyers figured out the details, but that was an eye opener.
Haplo12345@reddit
Can you name this magical PE firm? Sounds like a unicorn to me.
Bernie_Dharma@reddit
LVB Acquisition, Inc., a holding company formed by affiliates of The Blackstone Group, Goldman Sachs Capital Partners, Kohlberg Kravis Roberts & Co. (KKR), and TPG.
Haplo12345@reddit
Thanks!
HeWhoThreadsLightly@reddit
Like how do a company just forget 6 manufacturing plants?!
What happened afterwards?
Bernie_Dharma@reddit
We flew back to the US while the lawyers hashed it out. I rolled off the project early because my wife was in an accident and I needed to be home to care for her. Once they confirmed that the plants were in fact company property, they had a team fly back out to finish the job.
mschuster91@reddit
okay, now that's pretty standard...
wow, some PE with at least some functioning braincells left in management
wait what? There exist PE companies that actually give a damn?
tankerkiller125real@reddit
They actually surprisingly do exist; it turns out that there are millionaires/billionaires that do in fact want long term returns on their investments and not just short-term gains with long term bankruptcy proceedings.
Haplo12345@reddit
Generally those people act as individual shareholders, not as private equity firms.
tankerkiller125real@reddit
Where do you think PE gets it's money to make acquisitions and stuff initially?
InterFelix@reddit
debt layered on debt layered on debt. Their initial investments get leveraged by anywhere between 10x and 50x.
mineral_minion@reddit
Private Equity just means closed group of investors. Could be trying to bleed companies for parts, could be trying to diversify their holdings with successful businesses in multiple sectors. However, building a network of thriving businesses is hard, sucking the juice out of a business with a decent brand name is easy.
punkwalrus@reddit
One job I had, the CTO wanted me to go through their entire VMWare fleet and determine all the "dead systems," because more than three independent third party support companies were all competing with one another, and it was a goddamn mess.
The first round I eliminated 40% of the fleet because the machines were not even bootable. This was a significant cost savings in VMware RAM and drive space on the SAN. Next was a process to see if they were even bootable in a useable state, and another 20% were eliminated because they didn't have an active network connection and hadn't had anyone log in in years. The remaining 40% was harder because a lot of developers said that the systems were "essential to operation," but unable to explain why. Many said, "Our production database is on there!" and there was no database installed. Or no web front end for "an essential website relies on this." Those were hard to audit, but we whittled those down after a year because there was next to no network activity at all. I found "the scream test" eliminated a lot of those. Shut it down, wait 90 days, then remove. Only a handful of people even noticed. When I was done, over 1200 systems were down to maybe 200 actual systems that needed to keep running.
pdp10@reddit
That seems quite unusual for legacy servers. What were the causes?
punkwalrus@reddit
The problem was the VMs had some hard drive corruption, bad grub configs, "chmod - R 777 /*" by idiots, and some were sabotaged by warring contractors (like team 1 would mess up team 2's systems).
wyrdough@reddit
4,000 more devices actually on the network or 4,000 more devices sitting in storage that hadn't yet been disposed?
Secret_Newspaper2579@reddit
i tried to find out exactly what this guy said
it's a Linkedin live and i didn't watch the whole thing, but he does say 4000 devices on the network, yet not accounted for prior to the audit. don't know if it's bs, but that's what he says.
poizone68@reddit
I'll never forget how the security team at a previous employer proudly announced that their tool had discovered vulnerabilities in thousands of servers. This came somewhat as a surprise to the infrastructure team who only managed about 150 servers. The scanning tool they used didn't understand the concept of managed instance groups and horizontal scaling, and demanded the infrastructure team come up with a remediation plan for servers that only existed for a couple of minutes.
DesignerGoose5903@reddit
I mean the vulnerabilities should probably still be fixed if exposed, but stating every instance as a separate vulnerability finding is hilarious!
PixieRogue@reddit
Yeah, it’s really a garbage stat meant for shock value without context. Which, to be fair, is how most surprising statistics seem to be used. (Yeah, I know, I’m guilty of it with this statement.) 🤔
OneSeaworthiness7768@reddit
How else could they draw your attention to their platform being name dropped in the post? Won’t someone please think of the marketers
budapest_candygram@reddit
100% this
Sea-Aardvark-756@reddit
I've worked with enough blowhard managers and IT directors to know this is going to be whatever number was highest in whatever system they saw it in so they could make it into as big of a deal as they wanted to get buy-in for whatever step they wanted to do next. Probably registered BYOD devices exactly like you're thinking.
BezniaAtWork@reddit
Yeah we had a similar finding from an audit and had to explain that devices registered through MAM also appear and needed to show them how to properly filter out devices.
OneSeaworthiness7768@reddit
Yes but how else could they draw your attention to their platform being name dropped in the post? Won’t someone please think of the Brands
Gadgetman_1@reddit
Yeah. I have a truckload of old Cisco and Juniper switches and routers stacked in a basement room, waiting to be shipped off the e-waste recyclers whenever I can be arsed to move all that heavy crap.
And then there's the 'don't go there' shelf... with old PCs that for some reason or other has fallen out of the inventory system. Can't send it to the e-waste recycler because they send the management an email with a list of machines with brand, model and serials. and yeah, the stuff is branded with our logo.
The Drone Gurus have drones... Gods knows how many... It's not as if they stay in one place long enough for anyone to count them. Both drones and Gurus...
We DO NOT talk about RaspberryPi...
And if you mention MOXA, I will get annoyed... Really annoyed... Unfortunately, we can't stop a certain department from buying some very sturdy, very heavy, industrial measuring equipment. We're lucky if they actually ask us for a PC to run the control SW on.
Miserable-Scholar215@reddit
... That is going to be a pretty bad day for the people having to pay the licenses..
donalhunt@reddit
Oh Finance are probably just paying for 3X the number of licenses they are actually need. 🙃
bs2k2_point_0@reddit
Which falls on those approving the invoices, like dept heads. Otherwise that’s an audit finding waiting to happen for lacking segregation of duties and/or poor controls.
BezniaAtWork@reddit
I know of a few apps in our environment which are like that. I am just saving them for when we need to cut costs, as they never ask questions and the apps have already been identified and have been asked "Well just put it on the roadmap..."
bs2k2_point_0@reddit
I’m sure. I’m a director of finance for an npo so we are extremely cost conscious, more so than most for profit businesses. But I get how that works in larger corporations. I’m just saying it’s really not the fault of finance, or IT for that matter. The responsibility falls on the department heads who use those licenses and are budgeting for those expenses. Finance pays what is approved. IT manages the licenses. But it’s not up to us to determine if a department is buying double the licenses they require. We can certainly point out red flags, but it’s up to them to pull the trigger on changing how many are purchased.
If I was a cfo and found that my company was buying twice as many licenses as needed, heads would roll. And I’m generally not the placing heads on spikes kind of guy….
BezniaAtWork@reddit
Yep I have the emails where I've sent the notifications. We're double-licensed for Adobe Acrobat Pro. We were 2.5x licensed for Okta and I was able to save us $800K this past January on licenses there. We're over-provisioned for our E5 licenses by a good 50%. It's like 10 years of my salary every year is being spent on these extra licenses but it's never a rush to clean it up, and there's no incentive for myself to take the initiative as there's zero recognition from upper management for eliminating wasteful spending.
bs2k2_point_0@reddit
Wow. Suppose it goes without saying your leadership team is compromised of complete morons. I’m sorry man. Best of luck dealing with them. This one’s for you.
F
joshghz@reddit
It's okay, that's government! They have money from their revenue stream!
... oh.
SVD_NL@reddit
This is more of an access control issue than anything else. This is likely some department not properly logging hardware it gives out, or older hardware not properly being accounted for, and over time this stacks up. It should raise some red flags if you have way more users than devices (at least in groups of users that are expected to have devices), which is likely the case here.
The issue is: how were they able to access resources without hardware that was unaccounted for? Did they not enforce certificate-based VPN auth or Domain join requirements? Do helpdesk techs never notice they are troubleshooting PCs without RMM or MDM? Do they not check sign-in logs that show a bunch of unknown devices?
Security controls must align with procedures, and it's clear there's a huge gap here.
Kespatcho@reddit
A lot of government departments in South Africa have their IT managed by a state company called SITA. That company like all state companies in SA is famously corrupt and incompetent, this isn't a surprise to me.
jks513@reddit
Wouldn’t a corrupt organization provide one and charge for two and not charge for one and provide two? Seems like it’s harder to skim when you‘re providing double instead of half.
Kespatcho@reddit
I'm not talking about this particular instance, the organisation as a whole is corrupt.
daco_star@reddit
I spent 2 weeks consulting at SARS and I needed internet access in their building and an IT manager (I understand that they have many) instructed his report to sign into the WiFi on my Mac using his AD credentials. Wild. I turned down the offer and purchased a mobile router with a SIM card.
LizFromHexnode@reddit (OP)
Tier 1 support probably does see unmanaged devices all the time. But if their only mandate is "fix the connection issue" and there's no strict MDM/domain requirement enforced from the top down, they just put a band-aid on it and move on. It's a complete failure of policy enforcement.
Humble-oatmeal@reddit
More than MDM a asset tracking software is much needed I feel plus good practice of safekeeping
RangerNS@reddit
You don't know what the policy is. Maybe it is a stupid policy, or it lacks one.
Single-Virus4935@reddit
The father (50+) of my brothers exgirlfriend was CTO at one of a regional goverment office. Servers on the floor, no racks etc. Every morning he went to work at 5am to check if eversthing ia working. We never heared of monitoring tools. I showed him our icinga (or nagios, cant remember) and he was blown away. So I am not surprised.
Frothyleet@reddit
Stuff like that... some people just don't have the little smidgen of critical thought in their head that should make them think "surely there's a better way."
Like, so much of the skills and tooling I've picked up over time have been because I was taught or shown a workflow and thought "dear god that can't be the best way to accomplish XYZ." And then it wasn't, but no one had ever thought twice about it.
LizFromHexnode@reddit (OP)
That is both incredible and completely terrifying. If the people running the show don't even know what basic infrastructure monitoring looks like, there is absolutely zero chance they have a grip on end-user hardware floating around.
Single-Virus4935@reddit
They had conf*cker virus in 2011 hehe.
matt95110@reddit
During the pandemic the company I was working for lost track of 2000 laptops and several hundred phones. And you can’t use probably guess they hadn’t implemented an MDM.
retiredaccount@reddit
Unfortunately, even an MDM isn’t a viable solution for every device. Because besides laptops, during the pandemic where I worked, people took home second monitors, printers and other supposedly “essential” devices that were never recovered.
matt95110@reddit
Most companies don’t even track monitors and docking stations anymore.
cbelt3@reddit
Was it their hardware, or just BYO or IOT hardware on the network ? Still sloppy.
GardenWeasel67@reddit
It's not just the government. It's any large organization. We once found 500 laptops still boxed up in a storage room that had been purchased for a project that was canceled 10 years prior.
HeligKo@reddit
I don't know what country they are talking about, but I spent a solid amount of time in US Federal data centers. If the audit was using something like an CMDB inventory tool, then it is more likely that the IP scanning and the agent scanning weren't properly set up to reconcile and match devices found by each as the same devices. This would normally be a side effect of letting a vendor do the implementation and the person signing off on it being a bureaucrat and not a technical employee.
FatBook-Air@reddit
Some places don't have centralized IT purchasing, which is insane IMO but it exists. At my first IT job, anyone with a P-Card could order whatever they wanted and had budget for. Eventually, we did eventually have IT vet and order every IT device, but it took years to get there. It really took having bad budget years to get there.
tankerkiller125real@reddit
When I started where I currently work departments just got whatever they wanted, that changed after marketing spent 25K (a significant amount for the small business we are) on a marketing automation software, that couldn't integrate with our on-prem exchange server.
That got the policy changed to "All software purchases must go through IT" (which really upset a lot of departments, especially marketing who got railed in the meeting about the change). The hardware purchasing rules got changed shortly after as well when I dropped a CA policy requiring OneDrive, SharePoint, etc. only be accessible via Entra Joined devices. Talk about a clusterfuck that was when we discovered that a quarter of employees where on devices that had never been properly joined to the domain, and thus never joined to Entra.
These days, IT orders the hardware (billed to the department it's going to), ensures auto-pilot is setup correctly, provisions the device to the user who will use it, purchases the software licensing that's needed (billed to the department using it), and ensures everything meets our audit requirements.
FatBook-Air@reddit
Marketing can be the worst. In my experience, they'll pull the "welp, sales would be up but IT is hamstringing us" card. And if leadership knows no better, Marketing might get away with it.
tankerkiller125real@reddit
And that's what the CYA stuff is for, and policies.
RR321@reddit
Don't they have security scanners ?
OneSeaworthiness7768@reddit
This thread is posted by an affiliate of a security platform so yeah they’re trying to make that association.
mangeek@reddit
Yep. I was asked to add a 'data sanitizing' step to the 'hardware surplussing' process once and we discovered that only about 10% of the machines we purchased were ending up at that stage. Apparently it was an unspoken standard practice to just let staff keep their old machines when they got new ones, installed software, directory bind, and local admin rights and all.
I am the org's biggest party pooper.
raj6126@reddit
I’m trying to understand the other side of it. The procurement side. Where did the money come from to purchases double the amount of devices. Government money is usually appropriated.
Happy_Macaron5197@reddit
what get me is that this is a public sector org so those devices likely touched sensitive citizen data at some point. the question isnt just where are these 4000 devices, its what data is still on them, are they encrypted, who had access. an audit finding this gap is actually the good outcome. the bad outcome is never finding out at all.
derscholl@reddit
That’s what happens when you delegate inventory to functional managers and at the same time machines drop off domain for sitting in the drawer. Blue collar departments are already under so much pressure to run lean that this is obviously going to happen.
BisonThunderclap@reddit
I'm in MSP land. Inventory logging and tracking is hot potato between clients and MSPs, both want the other to do it and will do anything to avoid it. Haven't seen the machine in ages? Well, may as well assume it's stolen.
I'm still waiting for someone to invent a GPS device running on an emergency battery you can stick in or on laptops that will drop gps coordinates 7, 14, and 30 days after last being online. Would answer so many questions.
DaemosDaen@reddit
Used to work with an MSP who liked tossing that hot potato ... till they started charging for it.
Listed it under 'Inventory maintenance/management'. Available to entities under yearly contract. Got them a spreadsheet with their systems that we had in the RMM and that RMM pushed out via GP. Macs not withstanding.
The_Wkwied@reddit
We do client support as well as internal support, and we have hundreds of devices deployed nation wide. Every single device goes through our hands when it goes out the door, and when it comes back.
There's no other team involved. Just ours (and 3-4 people dedicated to client hardware.
When it happens that a device's status or location fall out of date, it is always due to oversight or laziness on our side. There's no other cooks in the pot. If we refurb a tablet and send it back out, but inventory still says we have it, but gps says it's in Nowhere KS, it's trivial for anyone to see who didn't do this.
We don't point fingers and raise hell. Remind the tech they forgot, fix it, ask them to revisit that ticket and double check the others. If the KB they were following isn't easy to follow, revise it.
Buuuuut when inventory and asset tracking involves multiple teams, and multiple can update and change the inventory willy-nilly, well, that's a bit anarchic with too many cooks.
TimTimmaeh@reddit
Depends on the category, but a 5-10% drift is NOT okay. Every untracked device with an user on it, not reporting back to compliance/patching/etc. is a potential high-risk asset and should not be allowed to the corporate network.
unstoppable_zombie@reddit
I've been working with a very large enterprise for about 15 months now on modernizing thier infrastructure. We've found over 7000 devices so far (switches, routers, APs, servers) that they did not have on thier inventory list.
We've been adding things with read only access to controllers and in every site/department is the same stories. "We thought that was decommissioned", "That must have been from acquisition X", "That's Bob's department. Bob: no that belongs to Steve. Steve hasn't worked there in 5 years."
estcst@reddit
First rule in government spending; why have one when you can have two at twice the price. -S.R. Haden
Reverent@reddit
Every government org I’ve audited has had the most piss poor asset management practices I’ve ever experienced.
Something about the bureaucratic nature leads to everybody trying to offload asset tagging and asset management to another team, forgetting the fact that they are the arbitrators of their systems and garbage data in is garbage data out. Then enter vendors who say they can “fix” bad asset management with enough fistfuls of cash.
MFKDGAF@reddit
RemindMe! 2 hours
lazyhustlermusic@reddit
That sounds removed from context just to give you shock value. Then they’ll pivot to being like buy our thing so you don’t miss half of your hardware, even though it could be spares or offline inventory
ALombardi@reddit
Shadow IT happens.
subWoofer_0870@reddit
Sh(adow)IT happens...
joshghz@reddit
Sh(ad)IT (hap)pens...ow
Bob_Spud@reddit
Seen the same in the financial sector. The company did a due diligence for an outsourcing contract and when they took control of their IT there were truck loads of stuff the wasn't included in the audit. The client got hit with a costly contract variation.
Ihavenoideatall@reddit
Legacy systems still in use?