something everyone's afraid to touch
Posted by Nexthink_Quentin@reddit | sysadmin | View on Reddit | 59 comments
everywhere i’ve worked has at least one thing that “works” but nobody really understands anymore and anytime it comes up the answer is basically “yeah let’s not mess with that." I like hearing stories curious what yall's are.
Rockleg@reddit
We have an app blocker on every machine and the blocklist is a spaghetti mess of parent process rules, grandparent process exceptions, wildcards for DLL paths, and so on. None of us are willing to try and cull the lists or simplify the exclusions.
I'm not really sure what we actually manage to block with all that but I definitely get tickets on the reg about the things we're failing to allow. My favorite is when the Intune team deploys a new version of an application and the name of the executable slightly changes.
The Venn diagram of the app blocker low-mode roster and the test group for the version update was a perfect circle was a circle. Now the queue is flooded with urgent complaints and the name is only a few characters different so you can't easily spot it scrolling through the deranged web UI for the app blocker. So you can kiss two days goodbye.
TxTechnician@reddit
AI chatbots are a perfect tool for that.
Ability to compare shittons of text in moments which would take us hours.
Rockleg@reddit
I get by OK with Ctrl+F, but getting this thing to export all the block rules at once is the tedious part. Have they invented a chatbot that clicks in six different places, waits eight seconds, and then repeats it on the next item down?
TxTechnician@reddit
So there's actually a Python library that will do a remote control feature with the full keyboard and mouse.
If there is literally no other way to interact with this library and you have to use a GUI.
You could totally have an LLM write a custom program that would do this exact thing.
The last time I used that Python library that I am thinking of was probably a decade ago And I had to build it for the exact scenario that you were talking about here where you had some old system that took forever to process and only had a GUI.
Rockleg@reddit
brilliant, thanks, I'll have a look. If nothing else it would be helpful to make a monthly archive of the rules
matroosoft@reddit
If you don't know what you block but are still investing time to support it, you're effectively maintaining an illusion.
Sounds like the perfect time to remove all and start over from scratch. First allow everything then slowly tighten it over time.
coolbeaNs92@reddit
This was the same at a company I previously worked at but I had zero involvement in the tool. It looked like it became (as you said) so borkerd and so many wildcard exceptions that from a security point of view it was probably pretty useless.
TxTechnician@reddit
I get hired by people to be the person who touches the thing that other people are afraid of touching.
ledow@reddit
The access control at my previous employer.
Based on a certain South-African system (ImPro), it was old and outdated. The main software ran on Java and had to run on a PC. But the software didn't work on Windows 11. So over time it ended up on Windows 10 LTSC (but, hey, technically you COULD run it on Linux!.... 2.2 kernel versions!) .
As time went by, they expanded and expanded but refused to change the system (too much "invested" in it). Also, by then, I'd made it work for almost everything we needed it to, so it was kind of customised and specialist and did things that the most popular "buy it and slap it in in the default config" competitor's products couldn't.
We maxed out the initial offering and the installers told us they could upgrade our main controller box (not the PC) to a more modern version that was backwards-compatible with the stuff we had. That allowed us to blow through a limitation on the number of door controllers, while still supporting all our existing door controllers, and using the same software (effectively).
It uses a Firebird database (which is a bit like SQLite - a single, flat-file database). But only Firebird 2, not Firebird 3.
It has an "integration server" that is a Java service that runs on the PC. The integration services lets you do some "smart" things that aren't part of the normal program. That was necessary for a "lockdown" functionality. If that service isn't running, a lot of stuff doesn't work even if the doors otherwise work.
I was asked to build a firelist - but the software module for that was AWFUL and it could only print out to an A4 printer (which takes forever) so we decided against that. I wrote one by directly querying the Firebird database. That expanded and expanded and ended up as two receipt printers that would - BEFORE THE FIRE ALARM EVEN SOUNDED - print out a full list of staff on-site, including those who had come and gone so we could check, in grouped alphabetical lists that could be cut at given points to give to the person, e.g., checked in Group A staff for a fire rollcall.
That script was INCREDIBLE and did so much stuff, so stupidly fast that the lists would print out before the first tone of the fire alarm sounded. People loved it. The people we used for access control wanted to BUY IT OFF ME for their other customers.
And then... I left. And my friend took over. And he managed it, because he knew all the ins and outs and what had to be running for everything to work, etc. Not in as much depth, but he could keep it running. But he hated touching my scripts without talking to me about it first because he knew he would break them.
And then... a couple of years later he left too.
And the guy they brought in to replace him DELETED all the notes on that controller computer, stopped all the Java services "because they were slowing stuff down", deleted all the documentation, and in his first day of being in charge - locked everyone out of the site. In fact, it was in the first few minutes. As my friend left site for the very last time, he noticed a light on the reader that indicated a service failure... and knew precisely what they guy had done. And he kept walking.
There is no "newer" or compatible system with the one deployed, and all the new ones are in the cloud, and they don't import any of the customisations we made. To replace it they would have to replace every single door controller, reader, etc. on the site - and we had already blown through their upper limits on the number of doors. They would have to buy TWO cloud systems to manage it all, and those systems would not integrate - you'd have to make the same changed to both every time. They would have no firelist functionality, after based their procedures around my code for YEARS (I warned them not to!).
Basically - throw it all in the bin and start again.
I warned them many times, and we documented it all extensively, and nobody - not the access control company, not the supplier, not the manufacturer, not the engineers, not even my friend much of the time - wanted to touch that system because they knew it would break something.
Well - it did. I would estimate... probably.... £75-100,000 to replace all that kit with the cheapest possible junk. And probably MONTHS of on-site labour.
Kamikaze_Wombat@reddit
Wow. Deleted the backups immediately. If I ran that business I'd be considering suing the guy for losses as he clearly lied about his skills and/or experience to have that job.
ledow@reddit
Same guy replaced the decade-old but up-to-date and functioning perfectly, with all ticket history, helpdesk system with some freebie thing off the Internet that was littered with ads because a) it was the only one he knew, b) he didn't know how to run a web server himself and c) he thought that just replacing it would force the employer to just let him buy the pay-for version that didn't have ads.
Literally a decade of asset inventory, ticket history, etc. down the drain because the idiot thought that would make everyone approve items on his budget that existed PURELY because he had no idea how to run anything else.
Did the same with MANY other systems. Basically "I don't know what that is, I'm not managing it, they'll have to buy a top-of-the-range replacement that I do know"... without a single check with everyone else.
Users literally had problems (so many problems!), went to the helpdesk like they normally would, and it didn't know who they were and there were ads in their face all over the interface. And that includes the bosses... who said WTF is this? And when he explained they took him into a private meeting and explained what change management, getting the user's consent, informing the users and budgeting were. Budget DENIED for all the above.
Visitor_X@reddit
Who was he related to? I could understand perhaps a week max, but a month?!
ledow@reddit
The irony is that my friend (who helped interview him) specifically told them to hire ANYONE ELSE but that guy, because he could tell even in a short interview that he was useless.
And then he warned them constantly while he was handing over that the guy didn't care, and was just planning on replacing everything.
Japjer@reddit
Seriously.
That goes beyond negligence and right into straight up sabotage. To delete all of that (software, services backups, and even the notes) really reads as more malicious than stupid.
BelugaBilliam@reddit
Holy shit. I enjoyed that story. Thanks!
stephenph@reddit
In my case it was an old IBM xt (this was 97 or 98. The power supply had died and it was under contract still so they wanted it fixed. I finally was able to locate the part (something like $400) and get it booted up. I asked what they ran on it and no one knew. I noticed it had a fax modem card so some investigating came to find out it was generating a report with no data and trying to deliver it to a disconnected fax number . There was one guy that kind of remembered it being a finance report of some sort but just knows that they were told to leave it be.
Poking around the files on the floppy yes a 51/4 floppy, it was generating this report and failing to send it for 6 years before it died. I was frankly amazed that the floppy was still functional.
TheOnlyKirb@reddit
The little PLCs that control very specific monitoring at a specific site that sometimes go offline if you hit it with more than 10 pings in 10 seconds
MegaSuplexMaster@reddit
We have an old DB server that has a custom application that was written in like 1990 they refuse to get rid of it, no one supports it anymore the company that made it is out of business so every time somthing goes wrong every points the finger in opposite direction "not touching that"
hkusp45css@reddit
That's crazy that a company would just decide to expose themselves to that level of risk. That's so shortsighted, I'd be concerned about their ability to pay me consistently.
mahsab@reddit
hkusp45css@reddit
It's not really my fault that you don't understand the difference between longevity and stability.
mahsab@reddit
And you decided it's unstable based on what? Because it's old it's "that level of risk"?
That's just some big words trying to sound smart.
hkusp45css@reddit
I decided it was unstable because if ANYTHING happens to it, the org is fucked, apparently.
If it never fails, it's not stable, it's fucking imaginary.
mahsab@reddit
Nowhere in the OPs post does it say this. They just say it's a DB server with an old custom app, not what it is for and how critical it is.
And things don't just "happen" to apps like a little mouse that will chew a piece of the code out of it or what? Nothing changes regarding this whether the application is old or brand spanking new.
hkusp45css@reddit
I'm going to guess you're in the first 3 years of your career. How close am I?
mahsab@reddit
About 25 years off
BoltActionRifleman@reddit
When I see this debate I’m reminded of this post about an Apple IIe still running a security gate at a storage facility.
Anlarb@reddit
English is a silly language, its just being used as slang for bad here.
Cricket_Piss@reddit
Ah yes, it’s impossible for a company to make any shortsighted decisions after 30 years of operation. Beyond that threshold, it’s pure perfection.
mahsab@reddit
It is not, but many people here have a knee jerk reaction of "this company is going tits up tomorrow" just because they have a vintage piece of equipment.
robjeffrey@reddit
If it works and they have backups.... why not?
I can spin a VM snapshot of an old OS/2 Warp every 30 minutes without impacting a .... ya... going to say it... a production server. If there is an error... restore and they've lost only 30 minutes of whatever it does.
If they decide to upgrade and it's baffed, we can restore any of the snaps and migrate to whatever or run whatever report etc.
It's network isolated, does it's thing, has backups and can run virtualized on whatever modern system we have kicking around in a pinch.
Basically kicking the migration can down the road until necessary.
RecoverLive149@reddit
Id like to introduce you to the US government.
TxTechnician@reddit
This is common.
You've got a company that has worked for years moving $$$$$ revenue. They run on some custom software (doesn't even need to be a program). No one messes with it cuz it just works.
Sea-Aardvark-756@reddit
Everything related to long-term non-IT vendor equipment is a huge mess, IT has a battle to keep systems updated but other departments don't even let the thought cross their minds.
CeC-P@reddit
The HVAC controllers. It's HP. It's XP or 7 or something. I just got a ticket for it last week. Ugh.
klauskervin@reddit
We have an windows XP machine running some sort of pavement analysis software where the vendor does not sell to private companies anymore. They will only sell to government entities. I don't touch it and take no responsibility for it so when it dies they need to figure out how to get the software again to complete their contracts.
Maelefique@reddit
The wiring inside an inherited building... all 3 previous installs...
Nietechz@reddit
It's like using Saturn-V to go to the moon. Technically we know how to go there, but probable no one knows how constructs more Saturn-V. (I know it's expensive).
PossiblePiccolo9831@reddit
sigh
Previously it was the solo developed dbase app that handled our legacy records (we work with a bunch of HIPPA data) whose sole developer passed away in 2017?
We were able to cobble a little report tool that touched the read only side.
Now it's this hellish custom CRM mess that is stuck on 2012r2 servers and still uses silverlight.
The best part... The silverlight code takes the files that are attached to the records (PDFs etc) and codes them into the db as base64 code. So the only way to pull the shit back out is downloading each file one at a time. That database has 10+ years of data for 1000+ people?
I am praying after I get it fixed from its latest debacle that it just sits there happy for the next 3 years. After that I don't legally need to retain the data.
AtarukA@reddit
Basically the whole infra.
Nobody dares touching it because if anything goes wrong, even if most things have gone right, the manager will be pissed off and on everyone's case.
Joestac@reddit
Sign engraving machine running locally off a Windows 95 computer. We just pretend it will outlive us all and leave it the F alone. It just works.
catnip-catnap@reddit
I work in OT (Operational Technology) so it's a long list: lots of industrial fixtures running XP, there's even a Win 2k PC and some DOS PCs. At least I managed to get them all air-gapped. All frozen in time because the custom hardware (PXI stuff, mostly) would be such an expensive pain to address.
Trust_8067@reddit
I try that strategy on all the equipment I manage, especially when we have to upgrade the OS.
ccsrpsw@reddit
Access 2010 has entered the chat :)
We have a number of machines that have Win11+Microsoft Copilot Office for Apps 365 (or whatever its called this week)+everything modern and up to date+Access 2010 runtime+some really ugly font hacks...
... to use some legacy ERP/MRP functions in a fully up to date, fully patched ERP/MRP system, that the vendor just can't get the new "doesnt need access 2010" feature over the line.
So every windows update, "reinstall font hackery" (its automated-ish now at least) if something changes in just certain parts of Office 365 or Windows that changes those 2 settings that mean Office 2010 runtimes can't find that one font it needs
nhpcguy@reddit
Access 97, running from a file share has entered the chat
ccsrpsw@reddit
Ugh - Win97 or Win95? :D
(And there is an MS-DOS 6.x machine that was just rebuilt a few weeks ago - but we know what that does and is done that way for specific low level HW access so thats a tad bit different than this scope!)
ipreferanothername@reddit
super important critical servers that are named after tv show characters because...nerds.
last year they complained about something dns related and i suggested...
sigh.
brokentr0jan@reddit
I have never worked at a company that did not name all of their computer and servers after characters and it genuinely drives me nuts. Worked somewhere where everything had to be named after a music artist. All the sysadmins who started this were ancient old dudes who liked naming them after classic artists so to mess with them whenever I deployed machines I named them after artists like Ice Spice and Megan Thee Stallion just to rile them up lmao
Arudinne@reddit
Got an old app server running SQL Server 2014. Fairly sure upgrading that will break it, but management doesn't even want to discus it.
It's siloed off into it's own VLAN with a very strict set of firewall rules around it.
SmartDrv@reddit
I won't say that these fall into the "nobody really understands it" zone, but it is wise to know your limitations on certain things and accept that it may be better to call someone who does before doing something you might accidently regret.
I'm a generalist (who seems to pick up on things to an oddly high degree), but there are a few areas I just don't feel comfortable touching.
-Legacy PBX (e.g. Nortel/Panasonic) or areas in IP PBX that feel more like Legacy PBX (e.g. Mitel Class of Service Options, Dialing Plans, Trunking, etc). If you really don't have a proper understanding, easy to mess things up.
-Alarm systems like DSC where you are banging in strings of numbers to configure things (especially when your zones aren't well documented)
-PLC program adjustments. I'm not trained, don't have Electrical background, on industrial equipment people can die from a mistake or you can cause severe damage/downtime. I'll help the person who does understand the program configure their serial to usb dongle and try and make sure they take a backup before making changes.
Some that are more relevant to the thread.
-Leftover SQL components on a Windows Server. Do you really need to keep some of those previous components from older versions of SQL on them? (Usually a backup and test situation on a less critical server or clone/test on something production if possible)
-Older runtimes e.g. Visual C++ You want to remove them to reduce patching footprint, but are they really needed? Same idea as above SQL.
-That Excel sheet with complicated macros/formulas that someone made who doesn't work here anymore that always felt held together with spit and tape. Sure you can probably reverse engineer it, but if you aren't a dedicated Sys Analyst role and don't really understand the numbers/its purpose sometimes you are best to say no.
In all cases, if your business depends on it, best to figure out a way to get control/documentation happening somehow before any catastrophe happens and you are stuck.
FnGGnF@reddit
Have an old sql server from 2008? Management won't let me touch it; even though we ordered fresh license and server for it last year for migration.
whatdoido8383@reddit
Way back when I worked for a MSP, they had a title company that relied heavily on a fax server for incoming and outgoing documents. It was a custom built server with like 14 fax cards in it and some super old fax software.
Of course no documentation and when you asked about who built it or how it worked, it was some old "phone guy" that left the country or passed away or something.
That was a real treat to troubleshoot when it started having issues and the owner was on my ass because half their faxes were failing in and out and they couldn't close loans or whatever.
They also used a Blackberry server for email to their phones... I had never seen one before or since, had to reverse engineer that one too.
SO happy I left that MSP without having to deal too much with that customer. What a duct tape mess.
rickAUS@reddit
On a similar note, at the previous MSP I worked at, one of our clients still used legacy call centre software that ran on XP. It was light weight, I'll give it that. But it was theoretically accessible from the internet if you knew what you were looking for and from there you might have been able to get lateral movement to the rest of the network. Took them 3 years to get new call centre software so that XP box could be ditched.
Hooray, right? No. Hell no. Because the survey software they used could only run on server 2008 (not even 2008 R2). Had to be internet accessible because they sent surveys out to people and they'd hit the on-prem hosted survey portal... yea, IIS on 2008 in 2022, great combo.
To my knowledge they are still using that same setup almost 4 years after I got a new job.
AZSystems@reddit
My heart ❤️
Hopeless_Wanderer27@reddit
Company I was contracted for had a separate third-party company come in and set up their security and distribution groups in AD. It was a nightmare having to sometimes search for hours to find the exact one a user needed/requested, but nobody wanted to go back in and rename them all and rewrite the descriptions.
hkusp45css@reddit
When I first got here, we had what I can only assume was the most hackneyed implementation of CUCM I had ever been exposed to. Nothing was configured correctly, or intuitively, or IAW any level of consistent logic, but it all worked ... mostly.
I want to preface this with the understanding that I am incredibly well versed in Cisco Collab configs and products, and this thing was only working out of spite.
It broke about once every 3 months in some new and novel way. All 3 of us would hop on it and start flicking switches and pushing buttons until it started working again. None of us ever discovered a consistent way to keep it working or to reconfigure it, so it was properly implemented. Not without completely tearing it down and starting from scratch.
Finally, we just shit canned it and went to VOIP in the cloud. Saved a small fortune, too.
Mobhistory@reddit
Exchange 2013.
Inherited it from previous ’IT’ group. Almost off it completely now but it was so hosed that it wasn’t uncommon for mail to be down for days apparently.
Took some doing to stabilize enough to even begin the process of migration to 365.
BrilliantJob2759@reddit
Oof, I feel this. We were v2007 at the time. Add in that the machine had hardware issues so we were afraid of looking at it wrong, let alone powering it down or even restarting. Backups sometimes worked, sometimes not. We had to contract out to some pros at an exorbitant rate (worth it) to migrate it over to a 2010 where we could then maintain & upgrade it ourselves.
Yuugian@reddit
The Amalgam server, that's not what it's called but, oof. I was hired just for it and it took years to tame.
Eleven different low-use functions with a PBIS/LWSMD/Beyond-Trust/Whatever-they-are-called connection to AD. Built on Solaris and migrated 8 different times. Perl and lisp and bash and kornShell scripts from 20 years ago in a bunch of different places. Processes that don't do anything any more. Processes that "can't do that" clearly doing that. Emailed reports and print jobs and batch processing and web dev and file serving.
Nobody that uses it wants it to change and nobody that supports it knows who uses it. It's been fun