Got pulled into a legacy cron job that sends SMS… with hardcoded vendor credentials
Posted by Fabulous_Bluebird931@reddit | ExperiencedDevs | View on Reddit | 72 comments
Someone noticed that SMS alerts weren't going out for account issues, so I got asked to check the old cron job handling them. I found a PHP script from 2016 with no version control, no logging, and vendor credentials hardcoded directly into the file, including a now-dead backup provider.
The script was still being called by a server that no one knew was even running. It silently failed when the vendor changed their api, and the fallback logic just returned true regardless of the result. No one noticed because the UI still showed “Message sent” every time.
I copied chunks of it into blackbox to figure out what a few functions were doing, and copilot tried to be helpful but kept autocompleting random curl examples that didn’t match the vendor’s API. I ended up rewriting the whole thing with proper error handling and pushed it into a repo for the first time.
feels wild how fragile some of the stuff we depend on actually is
madmoneymcgee@reddit
What I think happened:
Someone did enough to do a demo, after positive feedback from the demo they got another job and someone in charge just kept it up and running and whoever came in next never actually had to deal with it.
throw-away-doh@reddit
The number of times I have written a demo and that demo just goes into production.
Management see it working and are like "cool when are you deploying?"
PragmaticBoredom@reddit
I love all of these hypothetical explanations that imply it wasn’t intentional.
A decade ago, it wasn’t uncommon for something like this to be at the core of a business. As soon as the team gets it working they move on. They didn’t think about it again until it broke.
Many will be horrified, but look at the results: Someone put this together in an afternoon and it worked for 9 years. That’s how many small and medium businesses functioned on small teams of developers.
HoratioWobble@reddit
Oh my sweet summer child.
spelunker@reddit
Oh, POC? You mean version 1.0??
fork_yuu@reddit
My last job we released an app into a store and never talked about it ever again, the few that worked on it left. I still look at the reviews from time to time and it looks like it never got an update.
Fun times
dryiceboy@reddit
Sounds like 80% of the “projects” I’m usually left to deal with.
flavius-as@reddit
Funnier variation:
Right after the demo, they got promoted to staff.
non3type@reddit
I wrote a small automation on request from another department and left it running on a dev server for testing. Went on vacation. Code was given the thumbs up and I came back to find our production ServiceNow instance was calling it. It’s been years and somehow it’s still on the back burner, waiting to be moved.
throw-away-doh@reddit
We are like watchmakers except the watches we build are the size of buildings.
Buildings with no windows. Nobody can see in at all the intricate complications.
People just look at the time on the side of the building and observe "it works".
swegamer137@reddit
Sounds like the steaming pile of shareholder value I left behind at my first Shitshow Co.
SomeEffective8139@reddit
The best designed systems are the ones that keep running in the background so smoothly that people forget they are even there. Such a thing is beautiful to behold. The only problem with this one seems to be the error handling.
This reminds me... I have a theory that badly designed systems are rewarded in most software orgs.
If you build a bulletproof system that scales and is so well designed that it auto-heals when it falls over, there is nothing else to do and the system is forgotten, the developers get moved on. Nobody in management notices or cares how much excellent work was put into making this thing reliable.
But if you build a system that seems to work and hits all the deadlines, but is riddled with bugs and is a nightmare to keep running, this creates a ton of opportunities for improvements, bug patches, and fixes. Each crisis produces ways for someone to capitalize politically on the solution.
So a bady designed system produces more opportunities to demonstrate value than a well designed system. Which means that the organization is selecting for poorly designed software that just barely works.
Musical_Walrus@reddit
This is pretty much how all management thinks. Regardless of industry
genlight13@reddit
Soo kids, sit down and listen. This year i refactored a java batch job to generate some documentation from 1997.
It was originally created with java 4.
I rewrote it to use java 21.
Main problem for it was migrating it bc they still used Env variables for libs.
We have a lot of these kinds of batch scrupts lyong around. Main point why they aren‘t refactored yet is who got time for that?
We still use the rule „if it ain‘t broke, don‘t chnage it“
I am trying to craft some tickets for juniors but even the juniors get pulled away for some fantasy chatbot projects.
So yeah, i have a lot of code lines (mind the language) which were written before a lot of co-workers were born.
vvrinne@reddit
Java 4 came out in 2002. In 1997 it would have been 1.1
genlight13@reddit
Oh shit. You are prob. Right. So i am only the last in a line of rewrites.
Remark: the file date said 1997 and the code looked old old Java to me and i live with Java 6 Code ob my hands in one project. So i assumed that it would be 4. oh well.
mnk-9@reddit
I feeeel you, I've been rewriting old vb6 apps my company still runs on. The documentation for these are .doc files from the late 1990s.
originalchronoguy@reddit
2016? So 9 years. It had a good run.
CodeRadDesign@reddit
fun fact, the time between 1996 and 2005 was approximately 9 years.
originalchronoguy@reddit
Another fact.
These type of cronjob PHP mailer scripts that sent SMS was pretty common in 2005.. Mostly written by "citizen developers" with no formal dev background.
They found a stack overflow solution and copied and pasted it into. 0 * * * * /home/users/send.php
Not hating. But I've seen a lot of examples OP posted that followed this M.O.
KitchenError@reddit
Stack Overflow did not exist before 2008. The first public beta was September 2008 and it still took quite a few more years before it was really the place for finding code for everything.
csanon212@reddit
We had Expertsexchange and HotScripts!
d0rkprincess@reddit
Expert sex change? /j
shill_420@reddit
Hots crip t’s!?
hell_razer18@reddit
hahaha I like the term citizen developer as if devs have tiering classes 🤣
Filmore@reddit
9 years for a hack probably cobbled together in an afternoon? Heck yeah
ether_reddit@reddit
"sure, I can write you a prototype this week, but then I need a bit of time to rework it properly."
"no need, it seems to be working just fine, so you're being moved to another project."
trwolfe13@reddit
Our leadership pulled this move so many times on half-finished functionality that I actually had to get my team to start being less agile just to make sure our system stayed stable.
ether_reddit@reddit
I stopped writing proofs of concept because of this. Instead you get a document explaining that it could be done, but being light on the details.
The problem was is I was competing against a team member who was incredibly keen and started taking over writing these PoCs, even after I pleaded with him that it was just resulting in us ending up with more work and more legacy crap that we had no time to support.
hubbabubbathrowaway@reddit
Moving from Trunk Based to full on Git Flow because management can't stay on track longer than a day? Been there, it sucks, but you gotta keep the system running...
DagestanDefender@reddit
fragility of agility
NailRX@reddit
Sounds accurate. Happens more than you think
alppu@reddit
Given how devs rotate companies often, that's about three workers later.
DroppinLoot@reddit
I was going to say. 2016 is legacy. Damn I’m old
NUTTA_BUSTAH@reddit
Multi-generational at that point. Very good.
robberviet@reddit
Glad the company is still around to see it fail. Many don't.
YakApprehensive5334@reddit
I learned the hard way that when you take initiative by taking time to produce high quality code doesn't mean you will be promoted. So instead, I deploy half ass code that i was able to build in a quarter of the time that works just good enough so we can go to market quickly gets me a lot more respect and recognition in the organization.
ptolani@reddit
Honestly this seems like a story about how you don't necessarily need to apply engineering best practice to everything. This script was written cheaply and quickly and ran flawlessly for 9 years. I'd call that a win.
dhemantech@reddit
The script was still being called by a server that no one knew was even running. It silently failed when the vendor changed their api, and the fallback logic just returned true regardless of the result. No one noticed because the UI still showed “Message sent” every time.ptolani@reddit
Oh I read this, I just interpreted it as OP got involved because some actual issue was detected, perhaps in the order of days or weeks of malfunction, not years.
johanneswelsch@reddit
If somebody had spent an hour more for proper code and error handling for failed backups, the OP wouldn't need to have spent time for debugging and the business wouldn't not have lost the functionality.
There's no honor in garbage code. It's always a loss.
Fyren-1131@reddit
I guess realizing that people who write code are different, enables me to see that a bit differently. Maybe the dev at the time didn't know better. They might've come from customer service, or QA and had a knack for simple scripting. Probably hadn't received mentoring.
alanbdee@reddit
Reminds me of the time we had a printer stop working. Turned out it was a Novell netware print server in the closet and both hard drives had failed. It had an up time of like 9 years.
backfire10z@reddit
How many 9s of uptime is that?
pkmnrt@reddit
One
purpleWheelChair@reddit
🤮
imLissy@reddit
I fixed something like that recently, except it was a webhook for msteams, like a year old. Microsoft completely changed their API a few months ago, I guess there was a warning on the alerts, but I don’t get the alerts, the teams using them do. The vendor we are using to send the alerts didn’t know either. The calls were returning successfully and just not showing up.
Pagedpuddle65@reddit
Sounds like 9 years ago some did their job really well.
OntarioGarth@reddit
This reminds me of a repair that haunts me to this day. A reporting job stops sending reports. This code is old. Also we have no one updating this code, so I know I need to dig. Hours later I find it. The query checks a table to see if it should run. The table just contains two columns: month and year. It happens to be the month after the final entry in the table. I slammed my head into my desk. After I awoke I added a row for the current month. The next day I made sure the table wasn’t used for anything else, rewrote the proc to not rely on that table. The scheduled job can be turned off if we don’t want to run it anymore.
csanon212@reddit
We have this program which has a lookup table for years. Last January 1st we patched it at 2am when the first job failed by adding three more years and documented the hell out of it.
kronik85@reddit
hilarious
gulli_1202@reddit
how was the performance of blackbox compared than copilot and other ais
martinbean@reddit
Tell me you don’t know what it means to “blackbox” software without telling me…
No_Yogurtcloset4348@reddit
Nope, “blackbox” here is referring to Blackbox AI which I guess is some AI coding startup.
Check OPs post history; he mentions it in every post and somehow has a new story like this every day. Pretty sure this is just an ad for blackbox.
achthonictonic@reddit
Ah, you may have found the legacy of a BOFH. It grants +10 to uptime for unpatched, un-inventoried systems and services. It grants -10 to sanity. Looks like you made the right choice. Beware of etherkillers under forgotten desks or in the big box of random cables the server room/janitor closet.
ActiveBarStool@reddit
welcome to the real world buddy
SecondSleep@reddit
I had a very similar experience to this at a company you've definitely heard of. The product was an endpoint manager, and someone asked me to figure out what was going on with the system we used to deliver fix content to our business clients' networks. It turns out it was an un-source-controlled cgi-bin perl script running on an un-backed-up server. In the same directory were multiple, modified copies of the same script, named things like script.pl, script_modified.pl, script_modified2.pl, script_final.pl. People had clearly been in there before trying to figure out how the script worked, deleted and added logic, but had been too scared to delete the working version of the file, because it had no tests. I ended up source controlling it and containerizing the server, but with respect to brittleness, if that server had gone down, we would have lost content delivery, and endpoint management and compliance would have gone down across many fortune 500 companies.
PermabearsEatBeets@reddit
It's the XKCD comic within a company. https://xkcd.com/2347/
I've worked on some godawful legacy stuff that absolutely no one wants to touch and is powering some ancient api that can't be deprecated. Makes me shudder to think about it
rochakgupta@reddit
Oh hell naw
Goingone@reddit
At least the hardcoded credentials weren’t in VC
0ToTheLeft@reddit
or they were but the SVN server has been dead for years LOL
ParticularBag0@reddit
SVN “server” :’)
Last place i worked with it svn was on a smb share
Bootezz@reddit
They are now!
(Kidding, I hope)
AnimaLepton@reddit
Nice, my record is only 3 years.
effectivescarequotes@reddit
Your company neglected it for 9 years. That's not fragility. That's deriliction of duty.
Piisthree@reddit
That's a good one. I wish I could say it's the most janky script I've heard of in a production setup, but it's roundabout top 5 or so.
doberdevil@reddit
First time?
depresssed_soul@reddit
I feel you, when I try to explain this to my PO(who previously is an SME), just brushes it off lol,
And they cry when client drops support mails, i may have to try harder to explain how fragile that stuff is 🥲, but nobody is giving damn , i will try to keep the phoneix alive as long as I work here 😂 , but working on automating stuff on my own instead of relying on PO.
ItsNeverTheNetwork@reddit
This is awesome.
local-person-nc@reddit
Can't be an experienced devs post without shitting on AI somewhere 🙄
maria_la_guerta@reddit
bro how did you have an entire server running since 2016 that nobody knew about. Did nobody ask how these SMS messages had been sent for the last decade? Or have any observability into it?
Some things are fragile, yes, but this is less fragile than it is crazy.
brosophocles@reddit
What a happy ending, nice work!