Auditing is the bane of my existence.
Posted by valuablething123@reddit | sysadmin | View on Reddit | 58 comments
they have me in on a sunday for the physical asset audit
it's just me a tablet with a spreadsheet and a barcode scanner. walking through the cold aisles trying to match what the spreadsheet thinks we have with reality
just spent 20 minutes looking for a Dell R720 that, according to the database, is in this rack. it's not here. of course it's not. it was probably e-wasted years ago
five minutes later i find a mystery blade chassis humming away that isn't on ANY list at all. has no asset tag. no one knows what it does
i swear half my job is just being the only person who actually looks at what we physically own
Gainside@reddit
half the time the audit isn’t about compliance, it’s just the only time anyone actually takes inventory of reality
Beneficial-Wonder576@reddit
You're not a sysadmin you're an inventory analyst!
Breadfruit6373@reddit
My org has an entire department for this, ITAM. Those guys are built different let me tell ya
1C4R-@reddit
hmm
Sinister_Nibs@reddit
Why is this a weekend exercise? It should be performed during business hours.
beren0073@reddit
My question as well. How is this an emergency issue requiring weekend work, or are these normal hours for OP?
AlpsInternational756@reddit
One brought to mind the C-Suite won’t be there to railroad the audit at weekends. Seems like a very legit argument to do the audit at weekends.
Sinister_Nibs@reddit
C-suite should not be anywhere near the server racks.
InterrogativeMixtape@reddit
I do know OPs sorry but I worked in a smaller org heavy on MDM, it's easier to do a weekend sprint than track down mobile devices in flight during a work day, or break out mobile and non-mobile assets.
hasthisusernamegone@reddit
But these aren't mobile devices. They're rack- mounted.
retiredaccount@reddit
Welcome to the club. At my last engagement there were 83,000+ devices purchased over the last decade, somehow only 3,400 had ever been properly logged as ewaste. And only ~29,000 were actually still real; I had just started to untangle that knot when my engagement ended. Good luck to anyone who takes up the mantle next.
Okay_Periodt@reddit
oh girl... that is never getting solved ever. That would be a forever project at an org unless they hire on an external consultant or contractors to figure out that mess.
jhaand@reddit
If you do proper auditing, you wouldn't allow shadow IT to keep on running.
Enjoy the fireworks after unplugging the server.
CPUwizzard196@reddit
Time for a "Scream test"... power it down and see who screams
Okay_Periodt@reddit
Watch it be the hot heads...
AlpsInternational756@reddit
Call it a S.C.R.E.A.M-Test. Looks better on the paper. And if anyone asks: It’s called like that because if you power something down people will usually start to communicate in all caps and very punctuated.
Ssakaa@reddit
Probably not shadow IT, just carelessness, like noone recording what went to ewaste.
jhaand@reddit
I spoke to a CISO for a financial firm a couple of weeks ago. He was very adamant on confiscating unmarked servers running in a closet somewhere. Especially if they had a sign on them with: "Keep this running at all times."
Ssakaa@reddit
Oh there's definitely merit in it from a security perspective. Little raspi sized black boxes with a network wire tagged "do not unplug", especially. Or those older looking USB network adapters (turtles are fun) hanging on a rack plugged in somewhere with that.
Character_Deal9259@reddit
Ran into something similar with a client we had at an MSP I used to work at.
I was performing a physical security audit at their site, and was checking all their network closets and auditing their systems against what we had in our RMM.
Found a little NUC that was tucked behind one of the servers with a label that said "DO NOT UNPLUG! EXTREMELY SENSITIVE INFORMATION!" and it was plugged directly into their Comcast modem.
Found out that their IT Admin had setup a Nextcloud server with VPN and NGINX and was hosting photo galleries with illegal content on their network.
Brought the information to my supervisor, who got in contact with the CEO. Local Law Enforcement got involved, and eventually, the FBI too.
PsychoGoatSlapper@reddit
Thank you for making the world a better place
CySecJitz@reddit
Sounds like the audit is doing it's job as intended. A suggestion would be to pivot your approach and consider audits as a way to support getting things done. I.e dealing with stealth it or unmanaged assets.
lunch2000@reddit
The question is why does this need to be done on a Sunday instead of normal business hours.
krakadic@reddit
To prevent c suites and other executives from getting involved.
Okay_Periodt@reddit
At least then, this person can either flex their time or get paid overtime.
dustojnikhummer@reddit
Huh that is a good idea
CySecJitz@reddit
If that was the case, the title should be 'working Sundays are the bane of my life'.
zhaoz@reddit
For sure. Never let a disaster (or audit issues) go to waste. Use it as an excuse to get funding or change things.
jnievele@reddit
Or get things... Like that shiny server blade you found that doesn't belong to anyone as it doesn't have an asset tag. Nobody is going to miss it... As you need an asset ID to file a ticket ;-)
Gold-Antelope-4078@reddit
Yep this. If stuff is missing from the books or stuff is extra then that’s a problem. Once properly documented in theory the next audit will be a clean one. Although as someone else commented I don’t see why this would need to be done on a weekend they should allow the IT audit during business hours.
Okay_Periodt@reddit
This has always been a point of strain at my workplace. We are always finding so much stuff where we question where it even came from.
bQMPAvTx26pF5iNZ@reddit
Our higher ups demand its 100% accurate, which is impossible when staff move stuff without telling people.
Iliketrucks2@reddit
Lock the racks. Implement change control for hardware provisioning and de provisioning.
If you can get there then implement network controls. Are ports are off without manual intervention. At that point you make them provide all the docs you need, and update inventory.
Can’t do shit without packets!
MasterTater02@reddit
Sounds like a solid candidate for a scream test
DevinSysAdmin@reddit
Call your Executives and wake them up, state you have a unknown server on your network and it's a cybersecurity issue, unplug it, pull it out of the rack, do this 1-2 more times and you'll never have this issue again.
pdp10@reddit
Now you'll have worse problems.
sqnch@reddit
I envy your position massively, as an EUC engineer for a higher education institution that has literally no idea where any of its devices are - I’ve only just built a spreadsheet to capture most of the static PCs on campus.
Finn_Storm@reddit
I recently interviewed for a job with a 300 person international company with 0 IT infrastructure. Like, 0 zero. Supposedly they have an msp that manages a terminal server with their data on it, but when they have a new employee they will just go to best buy and get the cheapest non-chromebook laptop
knightfall522@reddit
Windows licenses? Office? Email? Do they even have a website? ERP???
Finn_Storm@reddit
Like I said they have some form of terminal server (and obviously a website), they may have other stuff but I wasn't about to do a full on audit in the first interview.
ProfessionalEven296@reddit
If you ever find a mysterious server, get as many details as possible and call the finance department. They should be able to tell you who authorized the purchase.
If you still come up blank, turn it off, as it obviously doesn’t exist. Then see who cries loudest.
headlesskid@reddit
Every audit turns into a scavenger hunt for ghost servers and mystery gear. Congrats you’re now the unofficial historian of your data center.
iama_bad_person@reddit
What year is it? SnipeIT self hosted is free, and if you want them to host it it's cheap as chips. Servers, laptops and desktops are tracked and assigned automatically and we hope to god the T2 boys put everything else in accurately 😂
nbtm_sh@reddit
I had to do this recently. For whatever reason, finance had 1 server as 2 boxes. I suspect it’s because an item on the PO was put as a separate line item. Regardless, I spent 4 hours trying to match the duplicate entry.
Ummgh23@reddit
Good thing we dont do that lol
UninvestedCuriosity@reddit
I started auditing any new hardware coming in years ago and disposals. Then at some point the parent org wanted to start actually bringing auditor consultants in. I exported a big spreadsheet and sent it off to them. They have been trying to get me to be a bigger part of their project to get everyone else doing it like I do it. The thing is. Nobody wants to do that and I'm not interested in arguing with people above me that they have to make it important. I'm happy to show any managers how we do it. They know the systems exist to do it and I've got a good process to make it less painful. So far nobody has shown up to ask.
Bullet was dodged. Don't let yourself get setup for failure.
datOEsigmagrindlife@reddit
Tell your employer to kick rocks if they want you to work on a weekend, something like this can be done during business hours or they can pay an external resource to come in an do this.
It's important work, but you're being walked over by accepting this.
Beneficial-Wonder576@reddit
Small shop guys think wearing all these hats is something to brag about. 😂
Vesalii@reddit
Unplug it, see who complains.
mcshanksshanks@reddit
the scream test
Isgrimnur@reddit
Mystery blade is obviously a security risk.
wrootlt@reddit
Oh, yeah. Had to do it at least once a year on my old job. But with an Excel sheet printed on paper (tablets were not a thing yet and later no money for such fancy stuff). Most things you knew were in place, but sometimes had to hunt a few things going through stuff in closet rooms, looking under the tables for that mystery switch. Software inventory was tougher though with lots of licenses acquired years ago with no records in IT, trying to find old scans of bills, to make sense where each license is assigned, do we have all VMs covered by our Windows Datacenter licenses, etc.
aaron141@reddit
I had to help with auditing when I used to work in the NOC for a bank
Spreadsheets then get
Asset tag
Serial number
Owner
Contract expiration date
End of life support data
Model
Name
And so on
engineerfoodie@reddit
I used to do this to. It sucked. As others have said more frequent (hopefully not on Sunday) audits can help. Then I changed jobs that was old cloud. No physical audits, no load calculations, no thermal assessments. Just focusing on configurations, automation, pipelines, IaC, etc. much better
gingernut78@reddit
As long as it’s double bubble, otherwise would be a hard nope.
Ashrayle@reddit
The auditors aren't responsible for you having a decent hardware inventory
arslearsle@reddit
Yes being the guy who can query and find things is always in need…
Creative-Type9411@reddit
to be fair whoever removed it looked at it too, lol