what's the dumbest thing that's been running in production at your company for years that nobody wants to touch

[-]

straightedge23@reddit

we have a raspberry pi under someone's desk that runs a python script monitoring our warehouse temperature sensors. it was a "proof of concept" in 2019. it's now the only thing alerting us if the hvac goes down. nobody has root access because the intern who set it up used his personal ssh key and he graduated 4 years ago. we just pray it doesn't lose power.

[-]

scheemunai_@reddit (OP)

the "proof of concept that became production" thing is so universal it should be a law of IT. i swear half the critical infrastructure at most companies started as someone's side project that accidentally worked too well to replace.

[-]

BigMikeInAustin@reddit

I try to push back on the "minimum viable project" once they find out I'm doing some extra logging, documentation, and in a team accessible place.

"Sure, can you put that in an email that you specifically don't want any safeguards or documentation? That if we use the MVP without hardening it, the business is okay with outages."

Sometimes they don't reply and I put in a half decent version.

Sometimes they do reply with an email and I stop all the "extra" stuff. So at 2 in the morning, I say, "Yeah, it was an MINIMUM VIABLE PROJECT and I can't remember how it works right now because documentation was not approved by management. I'll revisit this during normal work hours."

[-]

Magellena@reddit

Nice - and if I don’t write “minimum viable product” in all of the meetings and it turns into MVP…. I’ll get - hey - why does this important MVP service not scale?

[-]

BigMikeInAustin@reddit

Ha. Yeah. When they start a new project, they put some effort to say, “How exciting to be working on this MVP project! This MVP project is so critical. All the upper managers will be impressed when you get this MVP done! You want to be the MVP star, don’t you?”

[-]

jefbenet@reddit

Nothing so permanent as a temporary solution.

[-]

Magellena@reddit

Seriously how many pilot projects have become effectively “production” products … raise your hand if you’ve done this before 🖐️

[-]

kdayel@reddit

If it's just a python script, pull and mount the SD card, copy the script, and containerize it.

[-]

darthenron@reddit

At my last job I did an audit of some of the code that was in production. 30% of the code was created as POC or TEST function…

I think we ended up just renaming the scripts incase a customer wanted to peek under the hood.

[-]

Severe_Suspect_4002@reddit

Lmaooo

[-]

Final_Tune3512@reddit

lmfao

[-]

PauloHeaven@reddit

In case you really need to access it one day without wiping it, you can still shut it down for 3 minutes, mount the SSD or SD card on another Linux computer, generate a key pair and add the public key to the root authorized keys file.

[-]

No_Click_4097@reddit

I've done this before. I replaced my ssh keys and thought I'd updated all my connections. Deleted the old key. When I tried to reach a raspberry pi running a notification system I realised I had missed a device. Was quite easy to mount the SD card and update the authorized_keys file. And update my documentation.

[-]

EbbFlow14@reddit

Mount the filesystem on a separate machine and swap the ssh key with yours, you can fix your root access issue like this. Pretty trivial to do.

[-]

wh0else@reddit

Clone the SD card to a backup if possible. Or maybe just buy a new pi and make it a project to code a replacement with Claude and see if it matches performance, so you can then decommission the old one. Or at least have redundancy

[-]

Unexpected_Cranberry@reddit

Had a similar thing. Python (or maybe perl?) script running on a desktop sitting at a random desk. We managed to find it when finance started freaking out they weren't getting their reports when someone united it at some point. We just moved it to the server room and called it a day. We were all faced with either leaving or moving to the other side of the country when the company that bought ours were sitting down the office which was about six months away. Morale wasn't super high.

Those same reports stopped over more after some updates. Turned out the script relied on a database running on a virtual Linux machine that was running in a random file server. This was in 2006ish, so it was the only virtual server at the company. I think it ran under virtual box. We started it back up and let the guys at the main office know where it was. Noone had root access.

I wrote a super hack powershell script to deploy and update a metro app running on consumer tablets running windows 8 critical to our new fancy warehouse serving or webshop which has quickly become our largest store (retail chain). IT was not involved in choosing hardware or software. My boss asked me to see what they wanted and help when they were seeing up the new warehouse. I was handed 25 tablets and told they needed windows 8 and this custom built metro app on them. Go live is in two weeks. We've already tested with one tablet, works great. SCCM didn't support Windows 8 or metro apps yet. But MDT did. So I set up WDS+MDT share for deployment, and scripted the manual instructions the devs gave me fir the application. Took me about a week to get the tablets ready, I thought I was done. Next day the application needed to be updated. Turns out they were expecting several updates per week initially. So my deployment script grew to run on sign in (since metro apps are installed per user, and there were multiple users per tablet), reached out to a web server and checked if there was a new version of the script, if there was it downloaded it and replaced itself with the new version, then checked if there was a new version of the metro app on the same web server and if there was it installed it for the current user.

Super hacky, I'm not a dev. Was intended as a stop gap solution until SCCM implemented metro app support. It ran like that for eleven years at least. I know because a guy working as a consultant specializing in automation using powershell I know called me and asked how it worked. Apparently he'd been brought in because they needed to update something in the script. I told him basically what I wrote here and added the name of the webserver I had set up eleven years earlier. He called me back a few days later, thanked me for the help and reported the update went out without a hitch...

[-]

ek54ljl@reddit

Get yourself a copy of systemrescuecd or similar, flash to a USB stick, boot from that and change root password. You can password it in the short term.

Once you can get into the system, if it has visibility of domain controllers do a domain join. SPOF like that should not be feasible in even 2022. Check your co's release/approval/gate processes for putting kit into live environment, have policies changed if required.

Good luck.

[-]

sum_yungai@reddit

If it's a Pi, should probably clone the SD card first just in case that thing dies.

[-]

ModusPwnins@reddit

Yeah that needs to happen ASAP

[-]

reddit_pug@reddit

"nothing is more permanent than a temporary solution"

[-]

kuldan5853@reddit

it hasn't broken

That sounds like a good candidate for "whoops, corrupt disk. And nobody ever verified the backups. sorry, have to rebuild".

[-]

flammenschwein@reddit

Interrupt the network connection for a couple hours, get everyone's sphincters clenching, then suggest a proactive migration without actually disrupting business.

[-]

kuldan5853@reddit

Also a good way to handle it.

But to be honest, if it's really just a vbscript, it should be possible to transplant it to another host..

[-]

wtfisthissh1t@reddit

Key word being "should"

[-]

kuldan5853@reddit

I'd probably taken an image/cloned the drive/P2Ved the machine if it was physical, and then just tried in place upgrading the copy and see if it still works..

[-]

CuckBuster33@reddit

90% of the posts on this subreddit are these retarded AI generated question baits that get 1k replies each

[-]

Master-IT-All@reddit

This didn't seem that AI, unless AI is learning to write like a teen that didn't score well in Grammar and Punctuation.

[-]

CuckBuster33@reddit

They just prompt it to do that. The problem with that is they all still sound the same, just in a different way.

[-]

anonymouse589@reddit

A stupid invoice approval web app. We got rid of it 3 years ago but it was developed by a 1 man band that clearly didn't want anything to do with it anymore & charged about £700 a time to modify configs, add users or reset passwords. It would only open in internet explorer, edge's internet explorer compatibility mode wouldn't work, we had to add to the trusted sites list for it to even open. Reboot the box? you'll need to clear 2 logfiles then manually start 3 services in the right order else it won't work. To make it work the file permissions also HAD to be set to everyone, so all the PDFs of invoices going back years and the config files could be accessed by anyone who found the box. We tried "locking it down" to authenticated users, it didn't work. And those config files had the passwords for each user hard coded in plaintext, but you didn't even need them to log in and approve invoices - you just needed a username, which were at to the same as AD usernames to not confuse users, and the developer's master password, which was also stored in plaintext in the same config files, what was this super password? "12" . Yep, 12. Finance refused to upgrade to an alternative piece of software that wouldn't be comfortably at home in a gently steaming cess pit despite it showing as a massive red flag on every security audit. The only relented when we told them that it wouldn't work with Windows 11 and we finally killed it off, at least something good came out of that blasted OS.

[-]

dude_named_will@reddit

The XP computers running some sort of special ink software for our label printers. I still have to maintain a 2008 DC and file share because XP didn't like the newer stuff.

[-]

98723589734239857@reddit

what's preventing label printers from being upgraded?

[-]

Xaphios@reddit

Ah man that takes me back. We had everything from "this printer is provided by the courier and prints from their website after we book a job" to "these are whitelabelled goods and we label them for the end customer by connecting to the retailler's IT systems via their outdated connection software, but they're 100 times our size so we can't tell them to change it."

I had to set up a server 2008 standard VM in 2018 to allow us to decom the xp desktop in the corner running one load of software, and the server 2003 VM running another, and the wyse terminal provided by the retailer. All three companies said their software wouldn't work on anything that new, and they were all talking crap! Unfortunately we did hit a wall on trying to move to a 64-bit OS, but at least we could restrict access on that VM and only let it connect to the required IPs/web addresses.

[-]

98723589734239857@reddit

turns out i should be very glad we run Zebras that use software that's still being worked on... and i had no idea

[-]

dude_named_will@reddit

$$$

[-]

Zozorak@reddit

I had same issue, lucky for me it wasn't seated well and had an 'accident'.

Now if only the camera system that doesn't even record things finally break. That would be great.

[-]

Brilliant-Advisor958@reddit

I always thought it was more of people being unable to get across to management how screwed they would be when it breaks. Not if but when.

Often a good cost analysis helps.

Ie) If it breaks and we cant ship for a week, and we loose 1 million in revenue and piss off our customers all because we don't want to spend 20 grand upgrading our legacy system, seems dangerous to me.

That was a frank conversation I had with the owner once years ago. He changed his tune and was pretty supportive of spending when necessary after that.

[-]

DragonspeedTheB@reddit

Your XP machines should talk to a 2012R2 DC. I know.

[-]

dude_named_will@reddit

At least the 2008 machine is virtual.

[-]

ForPoliticalPurposes@reddit

A Panasonic Toughbook from 2008 who's only purpose is accessing the virtual timeclock software. It sits in an extremely dirty storage shed/steel building where our baseball field maintenance staff clock in and out.

It works. It has no internet access. The 3 old guys that use it know how to use it. That's all we know and all we care about. It will stay there until it dies.

[-]

waitingforcracks@reddit

No internet for a long time, the system clock must have a big drift by now!

[-]

AtomicXE@reddit

The dumbest thing running in production that nobody wants to touch? That would be end users.

[-]

jfgechols@reddit

I choked on my cereal in the break room. refused to explain the cause of my laughter

[-]

MechanicalTurkish@reddit

PEBCAK

[-]

Flashy_Resolution500@reddit

YOOOOOOOOOO

[-]

SquashNo7817@reddit

This needs a T-shirt!

[-]

ek54ljl@reddit

I have a tour t-shirt from overseas the grown-ups never liked.. "...here to fix what you effed up!"

[-]

CatsAreMajorAssholes@reddit

Printers and Quickbooks

[-]

BelugaBilliam@reddit

Program cards for PACS access on a tough book running XP.

HDD died last week. It had all the data on it. No backups. It's fucked!

[-]

BrainBlight@reddit

There is nothing as permanent as a temporary solution.

[-]

morningphyre@reddit

Needs to be a T-shirt

[-]

BuzzKiIIingtonne@reddit

We had an old freebsd 8 virtual server running up until we finally replaced it and the custom software it hosted last year. The thing was bullet proof, never had an issue with it since I started at least.

[-]

strongest_nerd@reddit

Someone joined the hypervisor to the DC which is a VM running on the hypervisor.

[-]

smilNwave@reddit

This is funny as hell

[-]

FrostyMasterpiece400@reddit

We once had virtual DC's depending on a Named nfs share bound to that DC's dns service.

It was a fun chicken and egg issue rebooting that lab lol

[-]

Final_Tune3512@reddit

this was back in 2008ish. Someone smart, smrt, built an access DB in access98 and had to where production could scan their RFID#s into it using the bar code scanner. Worked great until we updated all PC's on production floor to win7 64 Bit and the access db broke. So we had to rollback 300PCs+ to win7 32 bit. And still probably used to this day. I doubt anyone ever figured out that DB

[-]

nico282@reddit

Chain of scheduled script moving data from production serve, to intermediate server, to Azure files as an input to a PoweBi report.

I inherited it from a guy leaving the company, I passed down to the next colleague, that now has left the company too. Currently nobody in that company knows where, what, how or why it works.

[-]

Decantus@reddit

At my last job, we had a client who did Laser cutting/engraving. Their machines would only run off Windows XP. These machines would have cost $100k per to replace making it more economical to keep buying old parts off the internet and rebuilding these XP machines. No we couldn't virtualize as much as I wish we could have.

My boss kept buying parts off eBay and before that would hit up Weird Stuff before it closed.

[-]

cjcox4@reddit

Windows (any version)

[-]

texcleveland@reddit

if it ain’t broke don’t fix it

[-]

No_longer_a_pancake@reddit

Still use asp 3.0 for some things...no, not dotnet. I mean classic asp. As it that thing that got its last update more than 25 years ago.

There's also the perl script that sends email, because why not.

I swear it's held together by unflavored chewing gum and a monthly sacrifice of a virgin chicken.

[-]

RoyalMasterpiece6751@reddit

Server 2003 R2, nobody can really tell us what it does but all hell breaks loose in finance if shutdown.

[-]

MiKeMcDnet@reddit

The "big iron" billing system that has been around since the sixties (before I was born). They are remote hosting it and to do so they have an SNA to TCP/IP converter AND an EBSIDIC to ASCII converter. Slowly started to move off of it, but it's still around.

[-]

bluegrassgazer@reddit

I hope that 2012r2 box doesn't have internet access.

[-]

Magellena@reddit

Hahahahaha that would be …. Awesome!

[-]

0rphanCrippl3r@reddit

FileMaker

[-]

tshwashere@reddit

When I was working for a MSP, one of the clients is a machine shop that has a huge CNC machine that churns out oil platform valves. That machine is controlled... by a Windows 98 SE.

When the MSP took over one of the top tasks was to see how to replace that Win98SE. Virtualize, containerize, using a more updated OS to run the controller application. Nope, nothing works. The manufacturer of the machine has long since been out of business.

At the end we make sure the controller computer is air-gapped, and bought many replacement parts for whatever may fail and call it a day.

[-]

HispanOrtodoxo@reddit

Me pidieron valorar la migracion de un servidor de una terminal de descarga de vehiculos en un puerto maritimo, con software desarrollado por un fabricante de vehiculos europeo. Windows 2000 con una aplicacion en delphi y corriendo en una base de datos sql raruna de Borland. Los ficheros de instalacion dataron la instalacion en el 2000 y nunca la metieron un simple parche.

[-]

H0verb0vver@reddit

Your manager is a fucking idiot.

[-]

Time_Turner@reddit

How else would they be a manager?

[-]

dlucre@reddit

Yep. How hard could it be to replace that vb script, especially these days when gen ai has made this stuff so much more accessible to people without a programming background.

[-]

LookAtThatMonkey@reddit

Paradox DB running on an old once beige Fujitsu tower PC that apparently does something important to the shop floor person who goes skitz when it’s turned off. He won’t share its secrets. He’s three years from retirement thankfully.

[-]

og-golfknar@reddit

Act! Lol

[-]

mattmattatwork@reddit

Similar print server for a stupid label printer. Cant network the printer. Have a headless windows 8(?) machine that just serves the printer to a department. They dont want to foot the bill for a dozen or so more label printers (which honestly I thank them for). So this 'temporary' solution has been running for 5+ years with no monitor keyboard or mouse. Just power, network, and a label printer. Hostname, aptly named, junker.

I also have a ME machine that runs a medical device. We got it second hand 6 years ago and I'm scared to death of something going wrong, because I doubt I'll ever find the software again.

[-]

old_cypherpunk@reddit

Our accounting software suite consists of:

* Our main accounting software. Not even client-server, "multiple-user", designed for small firms consisting of a couple of people that the vendor doesn't even update or sell anymore. And it doesn't do everything the way we want so we also have:

* A separate Access database of spaghetti code that constantly breaks, corrupts and has to have eleven new fields created every time there is a change in partners. I'm not allowed to touch it, only my boss, the one who wrote it fifteen years ago.

* A desktop running Microsoft Money. I've tried at least moving them to GNUCash or Quickbooks but they didn't like it.

Lawyers. Deep pockets, short fingers. Ugh.

[-]

aibot776567@reddit

Point Claude code to that shit and refactor.

[-]

Fallingdamage@reddit

I use claude for a few items here and there. Just a bit of code or a CSV I dont want to generate myself.

Yesterday for fun I tossed it a seven page document and asked it to clean up and unify the formatting, spacing and headers in the document. It ran out up against my daily usage limits and when it finally finished the document was butchered.

Ill stick to giving it simple questions from now on.

[-]

sir_mrej@reddit

You need to put a picture on that server

I TAKE THE DATA FROM THE DATABASE AND GIVE IT TO THE ENGINEERS! IM A PEOPLE PERSON!

[-]

hardingd@reddit

All of it. Just kidding, only the most critical stuff. LOL. We’re making a concerted effort to modernize and replace a lot of technical dept. It can’t all happen at once, but we’re chipping away at it

[-]

Fnurgg@reddit

Server 2003 running a java 7 application... Don't ask...

I remember telling the team responsible 15 years ago that this stuff needs to be updated.

[-]

Adept-Pomegranate-46@reddit

An unscrubbed raid array.

Until it crashed.

[-]

Drew707@reddit

I just set something up that I am not exactly proud of, but it meets all of the requirements.

Me (former IT and ops director) and my boss (former COO) left our old company four years ago to start a consulting firm, but still are on retainer with our old company, me for IT and data shit, and him for client shit. Old company just went through a telco migration which required retooling the reporting aspect. My instructions were, "whatever you do, it cannot cost old company anything since they still pay us, but it also cannot really cost us anything."

So, that's why I now have a function app running in the consultancy Azure tenant that reads from the telco API and writes files to the old company's SharePoint and blob storage.

It's asinine. Manufactured herpes. I hate it. But it works.

[-]

Fallingdamage@reddit

"There is nothing more permanent than a temporary solution."

[-]

Master-IT-All@reddit

Customers with ancient gear I've seen:

Windows NT 4.0 Domain Controller, NT 4 servers to host a system that was built in the 90s and can't be updated because money.

[-]

Due_Peak_6428@reddit

if it breaks we'll deal with it then...sounds like a pretty awful strategy. do you guys ever have quiet periods?

[-]

ModusPwnins@reddit

This is the sort of thing that's easily fixable if they give you the resources to make it happen...

[-]

namelessmob@reddit

My boss

[-]

FauxReal@reddit

We had an RF transceiver being controlled by a server and backup server except nobody knew the login info for the software controller. I inherited management of it but could log into the machines but not the software. Luckily if the server got janky I could just reboot it and everything would start back up automatically. Eventually we switched to cellular devices over VPN to do the same job as the ancient Zebra Omni XT15 hand computers and decommissioned the servers last year.

[-]

NoradIV@reddit

Server 2003 running a custom label software, on a obsolete, unsupported ERP.

I also had to help engineering setup their automaton software on dosbox. The guy came to see me on a floppy dating from 1993...

[-]