It's friday. Experienced devs, what are your best war stories from a long career in tech?

[-]

bdemarzo@reddit

Not quite a dev story, but... A long time ago at a client far far away, I was a contractor working on a data migration project. We just finished, it was 10pm at night, when the client looked at his server's disk configuration. He noticed 4GB of "unused" space at the end of the disk, and created a new 4GB partition, and rebooted... and on reboot, mayhem ensued. The disk was a 16GB disk, and it already had a 16GB partition. Obviously, the 4GB "free space" didn't really exist, and why the client (who was a sysadmin) didn't think it was weird to have a 16GB and 4GB partition on a 16GB disk still puzzles me to this day. In any case, we worked through the night to recover data, and as a contractor, I took the blame, even though had nothing to do with it. Needless to say, it was happily the last job I did for them.

[-]

CorDharel@reddit

Well after a whole day of coding the mind starts to do tricks on you. It usually starts around 4-5pm.

[-]

bacmod@reddit

I was working on a "FUCK FUCK FUCK SEND EMAIL" with curl C lib and figured out that I can specify whatever the email address I want in the "from" field. That was a fun day in the office.
Our sysadmin didn't thing it was funny through.

$Million+ equipment with a (paraphrasing) Read/Write "public" "public" credentials. Again not a fun day for sysadmins.

I had to show up 8am Saturday one day in the field. Me and the CEO. And then listening to him bitching and cursing the field techs for about 4 hours. (we had to rewire everything).

I was once sent 700Km away in the field just to press a button haha!

[-]

CorDharel@reddit

Ah so you are that giy from the meme that needed to restart a server 700 km away hahs

[-]

gdinProgramator@reddit

Think I wrote about this here:

Unprotected API got pasted into slack for safe keeping, Slack tried to preview it, ergo called it a few times. It was a “bill all customers” API.

A few months and a few million to fix it.

[-]

santhastyle@reddit

GET endpoint to bill customers? Da fuq are you smoking over there?

[-]

Wodanaz_Odinn@reddit

A Post no Bills kinda shop

[-]

Kim__Chi@reddit

Got me so fucking good hahahahaha

[-]

anti-state-pro-labor@reddit

Genuinely fucking hilarious

[-]

GrapefruitMammoth626@reddit

Not just to bill customers.. bill all customers!

[-]

DuckDatum@reddit

CEO said, you know what would be really cool…?

[-]

GoodlooksMcGee@reddit

CEO learns to code, find out what happens next!

[-]

jakeStacktrace@reddit

Hey, once everybody agrees it needs to be a PUT and not a POST then we can go forward with that change.

[-]

2_bit_tango@reddit

And then frickin Cisco routers or whatever will update and block all put requests. Guess how long that took to figure out.

[-]

outsider247@reddit

Sounds like there is a story there?

[-]

2_bit_tango@reddit

Not a very exciting one lol. Multiple teams encountered the problem where some groups of users just…. couldn’t use put or patch. Worked fine from their home, on their phones, or different wifi, it was narrowed down to just some work networks hated anything but get and post requests for no obvious reason. Many devs trying to solve it off and on for over two years, infrastructure guys looked, web admins, peeps who knew network stuff, nobody could figure it out, IT on the users side did too. Requests just poofed somewhere between their computer and our servers with no explanation. No http statuses. Nothing. They eventually timed out or never completed if there wasn’t a time out. We “fixed” it in the meantime by just using get and post for everything. Gross. Turned out it was Cisco blocking everything but get and post on their firewalls or routers or whatever, I don’t remember what exactly it was, for “security.” So the work places using that just broke our app that used proper http requests. It was maddening, especially because there’s nothing we can do but keep using the bandaid.

[-]

gdinProgramator@reddit

Should be very obvious it was not a GET.

[-]

ings0c@reddit

Should be very obvious it was a GET. Slack isn’t making post requests to preview your links.

[-]

4InchesOfury@reddit

Wouldn’t the Slack preview be a GET? Why would Slack POST?

[-]

dbxp@reddit

Get Money

[-]

aztazm@reddit

rosebud;!;!;!;!;!;!;!

[-]

JoeHagglund@reddit

Maybe it’s cool to let AI take over actually…

[-]

sebzilla@reddit

Hah.. This is a great variation of the "URLs should never mutate resources on GET" lesson that many of us learned when browsers and plugin makers started shipping link pre-fetching functionality back in the early days.

Early versions of this were pretty indiscriminate and just visited all the links on a page, instead of ones specifically marked for pre-fetching..

Google Web Accelerator was one of the biggest culprits. Billed as a great way to make your browsing faster, it caused more problems than it solved.

We had a back office app used by our employees and suddenly we found records were being deleted all over the place and everyone swore they weren't doing it. Turns out the Delete link in our UI was just a GET request with a Javascript confirmation - that prefetching ignored - and the browser was visiting those links to pre-cache the results and triggering a (thankfully soft) delete.

Always mutate behind a POST request, people!

[-]

GeneReddit123@reddit

Some requirements make this not universally possible, though. Like a requirement to record visits or resource accesses in an activity log.

[-]

sebzilla@reddit

Sure, there's always edge cases or special cases, I didn't think I needed to be exhaustive here..

My primary point still stands that you should not put a mutation operation immediately behind a GET URL because that's not the expected behaviour of a GET request.

The REST spec had been around for barely 2 years when this happened to us, there was still a lot of "figuring it out" back then. :-)

[-]

JimDabell@reddit

There is no “REST spec.”; it’s not a protocol. The HTTP spec. has told people that GETs must be safe since 1996, possibly earlier. This was very firmly established before Fielding’s dissertation was published (2000), and certainly by the time tools like GWA came along (2005). The pain felt by unsafe GETs is down to individual developers not knowing, not the developer community at large not knowing. All the “figuring out” on this issue happened in the 90s as people gained experience with HTTP/1.0.

[-]

sebzilla@reddit

There is no “REST spec.”

Imagine walking up to people having a discussion about old times and war stories, and someone just uses the wrong word and you hard lecture the whole table about how they're saying it wrong and doing it wrong...

You must be super fun at parties.

[-]

JimDabell@reddit

Google Web Accelerator was one of the biggest culprits.

GWA wasn’t the culprit at all. GWA was following the standards; the systems that had a problem with that were not.

This affected a bunch of 37signals customers. DHH complained about it a lot. A whole bunch of people told him that he was wrong, the bug was his end, and that he should follow the HTTP specs. Instead, he tried to detect GWA to work around the problem. GWA came out with a new version and his detection failed, causing the same problem a second time. Then he started calling GWA “scary and evil”, while even more people tried to tell him the bug was his end. Just think, all that data loss because somebody couldn’t accept the way they had been doing things was wrong.

[-]

it_happened_lol@reddit

One of my favorite random exploits back when I was a young internet troll was to abuse forum software that did this.

Essentially, make a thread proudly proclaiming no one was allowed to respond and have a which would log out anyone attempting to reply.

Good times.

[-]

anonyuser415@reddit

Man, this is crazy. While Web Accelerator was before my time, there was some dumb prefetching JS plug-and-play script a marketing person dropped into our site via Google Tag Manager once upon a time.

We immediately got bug reports of frustrated users - turned out the mechanism began prerendering a page when a user mouses over the link. So when people were going to search, they'd on the way mouse over the log out button, and would be logged out behind the scenes.

[-]

Agent_03@reddit

Always mutate behind a POST request, people!

Or a PUT or a DELETE or a PATCH. All are valid uses of HTTP verbs in REST-ish API design, and generally make for cleaner interfaces than overloading POST.

Just never a GET verb, that should always be idempotent and non-mutating... and ideally have some cache-ability rules attached.

[-]

gaffa@reddit

Beautiful. Nothing malicious, no bad actors, just a classic small cockup with big repercussions

[-]

GraphicalBamboola@reddit

Not sure if having an unprotected API was a small cock up to begin with 😂

[-]

acommentator@reddit

It also sounds like a GET request was producing some a significant side effect?

[-]

Bozzzieee@reddit

That's actually a very good point. Had they used Post the disaster would've been avoided

[-]

campus-prince@reddit

Sarcasm? How would an unauthenticated POST be any better?

[-]

camelCaseRocks@reddit

Slack wouldn't have called it to get the preview

[-]

Bozzzieee@reddit

Slack cannot trigger it, for example

[-]

william_fontaine@reddit

Exactly. This would definitely have gotten people fired at any place I've worked.

[-]

DigmonsDrill@reddit

I was going to say "this could have been in the middle of development, and people need to be able to share broken incomplete things with each other over Slack", but ... why was it hooked up to production, then?

[-]

PmanAce@reddit

Imagine spending 30 minutes doing unit tests on it, on including auth unit tests on it...

[-]

dwight0@reddit

We had that same thing except it was get urls that performed deletes. Slack crawled them and deleted records.

[-]

TheGambler191@reddit

I will not scroll any further. Noone can beat this.

[-]

sol119@reddit

Somewhat similar. The sister team ran a load test - pushed a huge number of trades. Mistakenly pushed all that to prod url.

[-]

Attila226@reddit

That reminds me when QA decided to stress test one of our APIs that called a partners APIs. They didn’t mock anything and brought down our partners system. They weren’t too happy about that.

[-]

ikissgators@reddit

All customers' names updated to 'bill'

[-]

w-lfpup@reddit

A debug rabbit hole brought me to an 8 month A/B progressive rollout with hundreds of manager signoffs just to change 1 line of css.

It broke me. I am broken now.

[-]

tsaylor@reddit

Had a system that was writing a bunch of small files to disk, and another that would process and delete those files. We had monitoring in place to tell us if we ran low on disk space. One day we started getting errors like we were out of disk space, but the monitoring didn't go off. We had plenty of space left on the disk. That was the day I learned about inodes. Basically ran out of metadata space, so we couldn't create any new files. Always exciting to have things fail in novel ways!

[-]

donatj@reddit

Had a similar issue with a similar system. We migrated to XFS from EXT3 and it was never an issue again. System is still running now 10+ years later.

[-]

spacechimp@reddit

Years ago, I created the following TypeScript definition for values from what I thought was the most poorly-designed API I could possibly encounter:

type Foolean = 'true' | 'false';

Recently at a new employer, I created the following TypeScript definition for values from what I thought was the most poorly-designed API I could possibly encounter:

type Foolean = boolean | 'true' | 'false' | 'yes' | 'no' | 'Yes' | 'No' | null | undefined;

[-]

donatj@reddit

We work with a French company whose APIs return 'Oui' | 'Non' instead of a boolean

[-]

Adorable-Boot-3970@reddit

Had a CTO (a chemist by trade) in a hybrid department meeting scream at my team demanding a 10ms latency between UK and China.

And I really mean scream!

After a few minutes one of the senior directors, who though he was muted said, rather loudly, something along the lines of “it’s bad enough this fucking moron has never heard of common sense, but you’d think he’d heard of the speed of light.. ag well at least this is another team that knows he’s a total fuckwit”

The senior director was made redundant a few weeks later, but the CTO was gone a few weeks after that!

[-]

RandyHoward@reddit

I've been screamed at like that a couple times early in my career. I don't tolerate it any more. It's completely inappropriate in the workplace. The last time a superior screamed at me in the workplace was about 20 years ago, and they found my resignation waiting for them the next day.

[-]

donatj@reddit

Our DBA used to go off on us. He was let go for it, thankfully

[-]

EnArvy@reddit

In this economy I'd take a spanking head on and come back happy

[-]

robby_arctor@reddit

To be fair, you would do that in any economy

[-]

Askee123@reddit

If you leave a company due to that, what do you say when companies ask why you’re leaving/left that role?

[-]

RandyHoward@reddit

Simple... "I was feeling limited in opportunities for growth at that company."

[-]

lWinkk@reddit

I just straight up leave the call and end my workday. Has only happened to me twice though.

[-]

isurujn@reddit

And the senior director was hired as the CTO and everybody lived happily ever after, right?

Right?

[-]

haseen-sapne@reddit

right?

[-]

s0ulbrother@reddit

And they all got laid too!

[-]

EvilCodeQueen@reddit

Reminds me of the story of Grace Hopper using lengths of wire to explain to a pushy general how fast information moved in a nanosecond and why she could not make it move faster.

[-]

DuckDatum@reddit

Straight-line distance between the United Kingdom and China is approximately 7,799 kilometers (4,846 miles).

The speed of light in a vacuum is exactly 299,792.458 kilometers per second . In 10 milliseconds (0.01 seconds), light travels:

299,792.458 km/s × 0.01 s = 2,998 kilometers

Wow. If you used a super conductor on a straight shot—you’d still only be at less than half your requirement.

[-]

angrathias@reddit

The speed of light sounds quick until you break it down like this

[-]

ccricers@reddit

Sounds like they should've be screaming at their God to make it happen, or at least praying aggressively

[-]

ccricers@reddit

I wonder if there's a running joke somewhere that chemists are bad at physics and vice versa.

[-]

budgetboarvessel@reddit

This could work if India was still british.

[-]

DigmonsDrill@reddit

Sometimes you can get rid of a bad boss, but at the cost of your own job, too.

[-]

devhaugh@reddit

Lol love this. Hope the senior director got a new job.

[-]

donatj@reddit

Around 2012, we were a tiny company, I think 3 developers at the time and a server guy. We hire a second server guy and on his first day shortly after getting AWS credentials he somehow blows away our CDNs source S3 bucket... we just hear "OH ... oh no"

Took all day to get everything reuploaded.

He was with us for a couple years after that but a rough first day.

[-]

kutjelul@reddit

Joined a small team (4 developers) at a startup. Everyone had below 5 years of experience, except the 'senior' guy who had more than 10.

About 6 months into my tenure, the senior (technically in another feature team) picks up a small ticket, and (unbeknownst to me) spends 4 months on it. The ticket was renamed 'refactoring XX'.

Then, when I inquired, turns out he was practically rewriting the entire codebase, which was 2.5 years old at this point, because he deemed it unworkable 'legacy'. For some reason, no one in his feature team remarked that this was an odd, unplanned time sink.

After another month, our team sat down and he gave a presentation. I was baffled. This thing was 'ready', he said, but it was split over 12 PRs, all cascading, and while everything was new, everything was broken. Mind you that we were also approaching a time where our product would see a huge seasonal influx, and we were in a highly competitive market. The new code had no industry standard practises, instead it was a whole novel system that combined state management and dependency injection. If it were to be merged, we would have to spend huge amounts of time just to get back to a stable, releasable state. Getting any new people on the team would be an insane ramp up because it was so exotic.

The other people in my team were too intimidated and felt too junior to speak up, so I did. I suggested ways to move forward, such as merging one piece at the time and doing extensive testing so we can manage the risk. He promptly started sending me offensive messages and kicked me out of our 'work buddies' group chat.

That's when management got air of the situation, and took us into a meeting. To this day I still don't understand why, but it seems like management was high on 'self steering teams' so besides listening to our POVs, they still told us to settle it amongst ourselves. Maybe they were lost in the sunk cost of the senior developer spending months on something that wasn't releasable - but in the meantime, our high season was approaching and I insisted on not losing future valuable months of development because this developer didn't feel that gradual change is the way to go.

Then he campaigned with the management to have them not extend my contract (which was yearly and running out soon). Luckily they were reasonable enough to extend it anyway. I took the other developers on my team into a private meeting, listened to their feelings about the ordeal and the only conclusion was that they were intimidated and unconfident in the new changes. I encouraged them to speak up as well, and eventually the whole change was finally laid to rest. We continued to improve our project gradually, and we had no burning issues during our high season. The senior guy was so offended that he barely did anything anymore, and a few months later it was his contract that wasn't extended.

I still hung out with him outside of work a few times after that. Not a bad guy at all, it just seemed like he thought his seniority allowed him to break all the reasonable 'rules'.

[-]

ScrappleJenga@reddit

The dreaded pet refactor!

[-]

nazbot@reddit

Excellently handled btw. High IQ and EQ.

[-]

robby_arctor@reddit

Incredibly sus username, lol

[-]

nazbot@reddit

Shazbot was taken. I unfortunately didn't realize I'm only one i away from owning a electric car company.

[-]

robby_arctor@reddit

Lol, no worries, but that's funny af. Idk if you know, but "NazBol" is short for National Bolshevik, which is just Nazi by another name.

So you're pretty close in two different ways!

[-]

nonasiandoctor@reddit

If he was so senior you'd think he would know about the strangler pattern.

[-]

drguid@reddit

Once worked somewhere and a guy used a p0rn video for testing video uploading.

Of course the client saw it.

Always, always, always be careful when choosing test data.

[-]

Gupux@reddit

I've worked on a non tech company where the CEO wanted to build a product for the company. The product was being built solely by another (quite senior) developer.

The amount of features he output in just a few months was amazing, although the product itself was slugish, and there was basically no code quality, tests and documentation (which was completely understandable as the funding could be cut at any time). But this meant it was extremely hard to navigate this code base as a new (and junior) developer. It didn't help this other developer was a numbskull as a person and did basically no onboarding on a code base he was 100% responsible for.

There were multiple files with +10k lines which represented +20 related products, which went to almost 40 products by the time I was gone. Adding a new product meant finding every point with a huge `if` or `where` statement (which were multiple), and even go down the rabbit role of multiple functions with multiple individual `if`s according to the product type. To make things worse, not all `if`s/`where` clauses had all products, and there wasn't a single product that spanned over all clauses, so there was no reliable way to make sure you've accounted for all paths besides manually testing if the thing did what you wanted it to do.

But the worst thing was that since he was "the senior developer" and extremely obnoxious, for a few months I was afraid of asking questions. There was this one imported library that kept appearing that did a lot of things in the code base (encryption, database access, ...), which had a generic name and that I found no documentation on the internet. On the third month I finally asked where that came from (as again, he never said a thing about it). That's when he proudly told me that was a personal non open source library of his that he built over the years.

That meant that if he left the company, he could keep the company hostage with that code as it would be really troublesome to both debug errors and to change this library if needed. I immediately told my manager, which prompted my manager to immediately demand a backup of all code dependencies.

After I had access to the library code, I wanted to learn from him. Surely someone so obnoxious and with a library that did so many things should be a programmer macgyver. I started with the encryption function as I didn't have much knowledge on the subject. The encryption function was a wrapper (like most other functions in the library) that used a two way encryption function with a hardcoded salt in it (with no way to change this salt). This meant I now had access to most user passwords, but not all, as he truncated the function result to N characters, so longer passwords could not be reversed, which I actually verified could be done just to make sure I wasn't getting ahead of myself.

I left as soon as I could.

[-]

DeshawnRay@reddit

My first dev job in the 90s was on a high-pressure financial investments team. I was expected to code, test and deploy features and fixes within an hour or two of being told about them.

One time, on a Monday, they wanted a massive re-architecture and a complex new structure. They told me I had until Thursday. I remember thinking to myself "at least they've given me plenty of time to do it".

Absolutely blew my mind when I changed jobs and was told they have a 2-week release cycle, and that this was normal.

[-]

birdparty44@reddit

Quote from a never friendly unhelpful colleague on the worst freelance contract I’d ever endured:

“Once you understand how everything works, you won’t need documentation.”

Dude, literally the point of documentation. I basically was onboarding for 75% of the contract period.

Meanwhile, the codebase was a lot of poorly written ReactSwift code with equally cryptic unit tests to match.

They believed in “Uncle Bob” and created all these weird extra layers that just made the codebase like trying to walk through knee-deep mud, and believed in no comments as “code should be self explanatory”

Uncle Bob is US-centric where he doesn’t factor in a lot of ESL engineers and their highly ambiguous variable naming tendencies. 🤷‍♂️

In the end I’m now good friends with the CTO and he said I helped him understand the dysfunction of that team as they had a reputation for being “difficult gatekeepers”.

[-]

QuantumQuack0@reddit

ESL engineers and their highly ambiguous variable naming tendencies.

Lmao I know that feeling. Everything that triggers an event here is a "trigger", even the event itself sometimes. Both in hardware and in software. When you say "trigger" nobody knows wtf you're talking about here.

[-]

GistfulThinking@reddit

But the system configuration stands as documentation, right?

[-]

birdparty44@reddit

I don’t understand what you’re asking me.

[-]

tr14l@reddit

Had a guy commit interview fraud (multiple actually). But was an absolute rock star in the interview.

First day, I was all "he looked a little thinner on camera, didn't he?" But I didn't want to be the racist that can't tell Indian dudes apart, so I let it go.

Made excuses for over two months around problems with permissions, technical issues with his laptop, delays working with security and IT.

HR said we can't just fire him without running through a PIP and a paper trail. So, got a PIP together and suddenly his mom died, bereavement. Guy was able to delay stuff for SEVEN months and HR still wouldn't let me cut him loose.

Eventually, he said his family issues were too great and he had to leave to go home and couldn't work in the US anymore. 2 months after that, found out from a friend he was doing the same thing at another company.

I knew this happened, but was totally shocked at the scam and how ballsy and brazen it was. Dude got seven months of (pretty competitive) checks for being AFK most of the time.

Then, of course, the usual large data loss in prod, massive zero day exploits, screaming matches, one fist fight in an architecture review...

[-]

Scruff3y@reddit

Wait wait wait, a fist fight!?

[-]

tr14l@reddit

Yeah, well... More of a push fight and a thrown coffee mug. But... You get the gist

[-]

spline_reticulator@reddit

C'mon don't leave us hanging!

[-]

tr14l@reddit

It was a project manager and a tech lead. Project manager didn't seem to understand the "manager" part of his title was more "coordination" rather than "decision making authority".

That was kindly pointed out to him. Got heated. Project manager threw a mug sort of near the tech lead. Tech lead was a combat vet and wasn't about to tolerate that, so he shoved him... Some shoving back and forth, a blow was about to be thrown and our director barged into the room yelling that he was talking to a client and they could hear us. End scene

[-]

QuantumQuack0@reddit

Oof. Things have gotten heated here too but not quite on that level. Close though.

We have a new architect, lots of big tech experience and we're a start-up/scale-up, so immediately this guy is frustrated as fuck. In his first months I have heard so many shouting matches between him and the program manager, my manager, my skip-level manager. One incident where he threw markers across the room. He's toned it down a bit now, but still has to get his nose into everything. Half the company hates him by now. Problem is, he's smart af and we had a lot of trouble even finding an architect, so no chance he is going to get laid off.

[-]

gilmore606@reddit

Nice, I had a director of engineering throw a chair at me once. Dotcom days were a trip.

[-]

JackOfAllDevs@reddit

This EXACT same thing happened to us 2 months ago. But we figured it out and he was fired Within two or three weeks. We had two people try this at one time.

The first guy we figured out he was not a senior developer and let him go Within 3 weeks. Second lady was told in the interview that there was no work from home for the first 90 days. On day two, she went to my boss and said she was leaving for India for 30 days and have to work from home. Let go that day.

The red flag for both of them was wanting the credentials to remote desktop on day one even though they weren't allowed to work from home.

[-]

mcmaster-99@reddit

Wait.. you distinguished his appearance by size and not by his actual face? Im confused.

[-]

tr14l@reddit

Yeah, it had been several weeks since he interviewed (he asked for a later start date), so I couldn't really remember what he looked like too well.

[-]

danknadoflex@reddit

I’ve seen this bait and switch multiple commons. Common scam from particular parts of the world where the person who interviews is not the person who shows up at the job. I watched one get get fired in real time for this.

[-]

elusiveoso@reddit

I worked at a startup where we were given a pile of parts and tools on out first day and expected to build our cubicles.

I have been in the field for so long that ftp deployments were a thing an version control was a novelty.

[-]

bothunter@reddit

We had a site where users could upload photos. Previous dev just saved the photos using the filename that the client specified.

Anyway, someone figured out that you could just upload JPEGs with a bunch of pho code in the metadata and upload it with a .php extension. Since it was a legitimate image, it passed all the sanity checks and then ran the PHP code when we served it back to them.

Fun times.

[-]

ings0c@reddit

I joined a startup short-term loan company (payday loans but they were trying to shake up the market and make it ethical - no late fees, lowest APR on the market, stringent affordability checks).

My first day in the office, I got my laptop, and got set up with email access. I logged in and there was an email from the CTO saying the CEO had been fired.

Not a great start, but hey maybe there’s a good explanation, I’ll give them a chance.

Spoiler: things didn’t get better. The CTO had insisted on this frankly insane architecture.

It was “microservices”, but each service was essentially what would be a table in a monolith. So we had a Customer service, an Application service, a BankAccount service, you get it.

Necessarily, any user interaction required multiple entities, so a request came into the API gateway and would traverse many, many services, sometimes going backwards and forwards between them in order to satisfy the request.

This was slow and expensive. Bear in mind this was a startup, there weren’t that many people and the app could have been built by a competent Wordpress shop in a few months, the requirements were nothing fancy. The development environment alone was racking up £40k a month in cloud infra costs.

That crazy architecture was also, as I’m sure you can imagine, absolutely horrible to work with. It was like having a monolith with zero encapsulation, spread across the network. One service might update a property on Customer, and the Customer service just accepted it. You had no idea where updates were coming from just by looking.

So, development was very very slow. The board were becoming frustrated - why was this seemingly simple app taking so long to build? Are our developers just incompetent?

The pressure on tech leadership was building, and they thought maybe we could just move that pressure onto the dev team. They started setting totally arbitrary deadlines, without any developer input, and attached a fucking countdown clock on the wall.

Needless to say, that didn’t help.

Things continued in the same spirit and the writing on the wall was clear to me so I started looking elsewhere.

We did eventually ship something to prod, but it was buggy and unusable, and no one used it. We had £10m in investor capital, and the year I left the prod app made £3k in loans. 3k.

The company crashed and burned and a lot of my former colleagues were made redundant, luckily I’d moved into greener pastures by then.

[-]

Lower_Fish4403@reddit

Industry Experts: "Microservices could be a pattern for you if you want to break up that legacy, bloated monolith and delivery faster"

Architects and Executives: "LETS MAKE EVERYTHING THE MICROSERVICES!"

Industry Experts: "Uh... no, we were saying..."

Architects and Executives: "WHY THE MICROSERVICES NO WORK!"

Industry Experts: "Because Microservices weren't meant for..."

Architects and Executives: "NOW WE MAKE EVERYTHING THE LAMBDAS!"

[-]

Few-Conversation7144@reddit

It’s more a problem with their specific implementation of micro services.

A few facades and orchestrators would have solved it. Or a cleverly designed BFF.

Front end facing calls shouldn’t be aware of the microservice structure

[-]

csanon212@reddit

Microservices were this weird fad from 2017-2022. I was totally on board but only because hot well-paying startup companies were using microservices. I'll basically evangelize whatever tech the highest bidder is using.

[-]

Lower_Fish4403@reddit

It was ideal for certain cases where legacy systems needed to be modernized in mid to large enterprises. Folks like Fowler always said it's just a pattern, not a religion. Then it became a religion. I've seen Microservice architecture help an organization scale as they grew from a $4B to $10B B2B. I've also seen Microservice architecture completely hamstring a $2B organization with massive service sprawl, duplication, tight coupling, etc. These days, if I walk into a gig and see the "customer-contact-address-update-and-delete-service", I know it's going to be a lot of long hours.

[-]

spline_reticulator@reddit

The app itself cost £19m?! But how?

[-]

ings0c@reddit

Salaries are included in operating expenses 🙂

The execs were very handsomely compensated, and dev teams are cheap!

[-]

nighhawkrr@reddit

Someone accidentally put an extra 0 in my day lenders amount for loans in a release.

[-]

rudiXOR@reddit

Sounds like a typical "Startup-Microservices" experience.

[-]

ings0c@reddit

It’s a shame. The dev team were actually a good bunch; there was unanimous support for going down a more event driven route with larger services, but the CTO was having none of it.

Oh, I forgot to mention that there was an architecture team that sort of designed this hell at his behest. There was a giant data model diagram that was updated every time new features were planned.

I think the CTO might have just been a bit dim, and thought the logical data architecture should map to physical servers one-to-one. I really don’t know.

There are some tenable arguments against eventing, mainly that it’s harder to hire for, and requires some care (though I don’t agree these are good reasons). I don’t think that was his thinking though.

[-]

sebzilla@reddit

there was unanimous support amongst the principal devs / staff / tech leads for going down a more event driven route with larger services, but the CTO was having none of it.

I always find these kinds of stories strange, because I wonder why the rest of the senior team didn't take their concerns to the board (particularly the investor seats) or to the other C-suite leaders...

Unless the CTO was somehow the biggest shareholder or was best friends with the investors, it seems like they would have at least wanted to look into it, had they known...

I'm not backseat driving here, hindsight is a luxury you don't have in the moment.. but yeah, always a bit of a mystery when I hear that one person's technical windmill takes down a whole company full of smart people.

[-]

ings0c@reddit

I can’t speak for the others, but I don’t care enough; I’m paid to do a job, not pick fights with the C-suite.

Don’t get me wrong, I care about doing my job well, but I’m also not putting it at risk by trying to take down people higher than me in the food chain.

I was also a contractor (in the UK), it would have been very easy for them to can me if they liked.

I’ll argue the pros and cons of a technical design, and be fairly persistent to get what I think is the best outcome, but if I get a firm no along with hostility, I’m not doing more than that.

[-]

beefz0r@reddit

Not my story, but back when everything was still on premises, there was this machine that went down every morning and nobody could figure out why.

Turns out the cleaning lady pulled the plug briefly for her vacuum cleaner

[-]

DisastrousFruit9520@reddit

May favourite stuff is requirements that run into hard limitations of the system they're for and you just have to do it anyway. I have two stories like that:

When I first got into the games industry I had to remake the UI for a HD update to a classic game. However the original plan of replacing the UI renderer with an up to date one fell through and I had to had to make the UI using the very old system. One of the limitations of this system was that it didn't support Arrays. This was Actionscript v1, so my only recourse was to create a number of empty movie clips on the canvas with names like "box-tween1, box-tween2" etc, then use eval to grab them in a loop and store values on there. Worked, but was such a hack.

Second story was when I worked on the kitchen and POS system for McDonald's. We had a requirement to allow messages to be sent between POS terminals like a chat. The system didn't have any kind of asynchronous processing, but it did have a remote execute function where you could trigger a function call on another machine. Problem was that although you could use that to send a message, it didn't work if you were in the chat because of how the system worked; the remote message would be received but there was no way for it to write directly to the screen. My solution was to use the remote execute as a kind of event loop where I would execute a remote function locally, targeting itself, this would put the remote execution in the back of the queue, so behind the one that was included the message. This meant the message could get written to a buffer then the local RF call could flush the buffer to the screen. Very pleased with that one.

[-]

IrishPrime@reddit

A coworker was iterating on some infrastructure as code. In order to make sure it all worked correctly from scratch, they were tearing down all the resources and creating them again. Edit, create, confirm, destroy. They had done it enough that they got a bit complacent. Eventually, they destroyed the wrong infrastructure.

Everybody started getting alerts and reports of services being unreachable and my coworker came over to me and told me they just fucked up big time.

We went straight to the department head and told them that we knew the root cause of the widespread outage and explained what happened.

We had to rebuild a huge part of our infrastructure (a VPC and basically everything on it) and had our team working 'round the clock through a weekend.

It was crazy, but it sure helped us finish getting a bunch of that other infrastructure as code ironed out.

[-]

DarkyHelmety@reddit

Theee was that time when I was hired as a senior developer and not give a computer for two weeks..... Hey at lwats they printed some docs so I could onboard but wow, not impressed. This was not a small cie either but my manager was useless.

[-]

unixmonster@reddit

That time I was remotely working on a firewall that immediately applied rules when you typed them in and accidentally forget to allow before you deny all.

[-]

LoudAd1396@reddit

My two favorite stories (same job)...

STORY ONE

Client reports problem with the website. I ask for a screenshot. They send an excel file...

With a gif in one cell...

Of a fax to themselves....

Of a photocopy...

Of a laptop screen face down on the copier

SECOND STORY

Doing a web site for a Greek women's charity organization. Got a virus off an image file they shared.

It was a Trojan

Beware of Greeks bearing GIFs

[-]

mello-t@reddit

Here is one. Worked for a large (not Amazon) e-commerce site back in the day. Tech stack was being moved to Java but the legacy system was c based using shared memory segments and semaphores with a backend daemon doing all the logic fronted with apache. About half the site was migrated but we still had a lot of traffic hitting the c site. Some nasty bug with semaphores not getting released (good ol c) causing the shared memory to start bleeding between sessions. Almost no c programmers left in the company and a few days till Black Friday. Literally saved Christmas by writing a Perl script to clear out the semaphores and reset the memory segments every minute and ran this way through December.

[-]

the-creator-platform@reddit

launched a startup. was mostly nothing til tiktok got banned then we suddenly received constant traffic. i had put together a basic autoscaling policy but wasn't prepared. wrangled with scaling all day until it was running smoothly. was something like a 14 hour day, was about to log off, then tiktok api got cut in the US. Suddenly I had to migrate our app servers to another country while keeping data stores like pg and redis put (it was too much to migrate them given every minute was wasted). Luckily, I had defined the whole system in terraform so I was up and running in the EU by morning. Then Trump announced it was un-banned and it was all for nothing :D

But... that constant debate in bigger companies about "we need IaC because what if our datacenter goes out" is real

[-]

belkh@reddit

It's really more than just our datacenter goes out, but it's a big help as well! Being able to spin up recovery, new preprod env, automated testing env or custom client deployments is super useful

[-]

Perfect-Campaign9551@reddit

I think I understand some of these words

[-]

the-creator-platform@reddit

Of course exactly

[-]

danielt1263@reddit

Worked at a gaming company and we had a deadline that required us to work stupid hours for three months straight (12hrs per day, 7 days per week). Finally finished the project on time after working through Labor Day weekend. On the day after delivering the game to the publisher, the CEO comes out of his office and fires all the 3D artists. This started a literal brawl in the middle of the open-office area.

The gaming industry was absolutely nuts back then. I hope game devs have it better now.

[-]

danielt1263@reddit

I was working on a C++ 2-D gaming engine for MacOS 7 as my first professional job. MacOS did not have protected memory until OS X. It was quite easy to brick the OS accidentally with any memory or stack overflow. I'd have to unplug the computer and plug it back in to recover.

It would take weeks to track down almost any bug.

[-]

funbike@reddit

tl;dr: For 30 seconds I thought I had destroyed a startup.

In 2000 I was CTO at a tiny startup of 12 people, with 5 programmers and 1 sysadmin. I found out the sysadmin wasn't checking backups and the backup process hadn't been working for almost two months. So, I came in after hours to fix the backup process and do one manual backup.

So, I ran the backup process. But it appeared that I had arguments reversed and I had accidentally overwritten production data with a two month old backup.

This data was time senstive, so going back that far made it nearly useless. I felt faint and got down on one knee. I thought to myself, "I can find another job. Maybe I'll have to move, because this city is too small for this not to get out. Can they sue me?".

Then, I looked again to be sure. "Whew!", I thought. I actually did it correctly. Since then, I've always insisted that two people be involved in any work on prod servers.

[-]

deer_hobbies@reddit

I got my "wizard status" at my first company for refactoring a N^4 nested for loop that took around 3.5 seconds into a 12 line SQL query. The SQL query was gnarly, but it was one of the silliest code files I'd ever seen. The only reason the for loop even completed was that they made a fresh DB table per tenant every 6 months.

[-]

s0ulbrother@reddit

When I joined my current company and one of the first things they complained about was Django being really slow. I installed toolbar and will you look at that 500 queries per a page because they didn’t know to prefetch.

[-]

s0ulbrother@reddit

I nearly took down a major insurance company by missing a nolock on a sql stored procedure. And by take it down I mean claims reporting.

In my defense there are a thousand ways this shouldn’t have been on me, it went through code reviews and testing environments. I was a junior and put in a role I was not remotely trained in.

[-]

Breklin76@reddit

About mid-career (currently 27+ YoE), I was the sole full-stacker for a large casino’s Marketing Department. Our E-Mail Manager stopped coming in and there was a tiered member blast that needed to go out ASAP.

I had no working familiarity with the actual campaign configuration. I only worked with the email template component one other time. I built the emails once in a while and would import them for the missing manager. That’s it.

Management comes to me and says we need you to do this. So and so has the data sets you’ll need for each tier, you just need to map each set to the appropriate email promotion. The data sets were for each players club tier.

I ended up sending a 1000 base tier players an invite for a free lobster feed.

[-]

pseudo_babbler@reddit

That's quite a wholesome outcome from an error though. I hope they all came in and enjoyed their lobster.

[-]

Breklin76@reddit

Actually, we caught the batch at around 250 so the collateral damage was minimal. They had to shell out some clams for free play. Mistakes happen.

[-]

gilmore606@reddit

had to shell out some clams

I would have demanded the lobsters tbh.

[-]

Ok_Concert_518@reddit

using a system which worked very very very well and all cloud integrations also worked well and lots of hours spent setting all those up.

customer support of the system is amazing.

IT decides they want to replace everything with a "cheaper system"

im pretty sure some execs making bank with this decision. they even hired consultants to work on the "integrations" for the new system.

i could not see any reason why to stop using our current system. its scalable, its really the best. not top tier, not expensive. it just works.

[-]

sneaky-pizza@reddit

Consulted to a very toxic startup. The last step of their hiring process for devs and product was to bring them to a session with the CEO who would apply “intense pressure” to see how they handled it.

[-]

spline_reticulator@reddit

The interview process

https://www.youtube.com/watch?v=da4PhXJzDcs&t=4s

[-]

anonyuser415@reddit

this is just a normal hiring process for anacondas

[-]

riversilence@reddit

This one time, in band camp…

[-]

dbxp@reddit

We had some Ukrainian contractors who took a half day to relocate during bombing runs and then carried on working like normal.

[-]

spline_reticulator@reddit

✊ 🇺🇦 ✊

[-]

Complex-Magazine6690@reddit

I worked for a company with a tonne of Ukrainian software engineer contractors who really were treated more like a core part of the team, there was a great relationship built up. When the war broke out, the company told them straight up "work if you feel like it, but we will keep paying you no matter how much time you take off" and despite such an offer, they still kept working the same hours except for when they literally had to run to bomb shelters. The ethic of these guys!

[-]

reddit_man_6969@reddit

I mean, how long is that offer really good for. For sure they would have stopped paying by now. Smart Ukrainians nurtured their income stream

[-]

darksparkone@reddit

That's how we are taught, guess it's an echo of soviet era propaganda. Taking a day off when you are ill but not bedridden feels like a cheating (and big modern companies still struggle to keep ill people home and prevent a bigger virus outbreak, while old-fashioned companies are truly mad if someone doesn't show with "just 38⁰C" and a flu).

And we're really grateful for colleagues and companies that supports us during this hard time, it means a lot. I know the motto of career advice subs is "business is not your friend and will dump you on the first opportunity" - but business did keep with us despite being a risky asset, and supported us, and it genuinely feels like a little miracle.

[-]

belkh@reddit

Work helps distract you from the ongoing anxiety

[-]

Eyedema@reddit

I had some experience with a couple devs from Ukraine and can only say good things about them. Great people and admirable work ethic. Kept working even when shit hit the fan a couple years back!

[-]

anonyuser415@reddit

For 4 years I worked with two Ukrainian contractors who each lived on opposite sides of the country and we all slowly realized that their PTO kept coinciding. They wound up getting married. I know he stayed to fight in the war, and hope he's alive.

[-]

stevefuzz@reddit

Lol I loved the Ukrainian devs I worked with! They were hilarious and really easy to work with.

[-]

DuckDatum@reddit

I used to work for this company that sold online storage, as well as a marketplace for secondhand goods because the CEO said they should, because he’s able to turn a profit flipping medicines and crap off eBay, so the business should be able to scale that—right?

Any chance that sounds familiar?

[-]

lunacraz@reddit

worked with a couple at my last gig they were great

[-]

kenybz@reddit

Literal war story

[-]

GeneReddit123@reddit

Some 15 years ago at the start of my career, we were working with a remote contractor from a post-Soviet state. One day he sent an email that he can't attend work today, because it's his turn to join a local militia patrol, and sent a photo as proof - AK-47 and all.

[-]

dingdonghammahlong@reddit

Ukrainian devs are awesome, I have had the pleasure of working with a few and they were the best devs I have ever worked with

[-]

DigmonsDrill@reddit

"This is why we have flex time."

[-]

Shanix@reddit

We have a studio in Kyiv, and the whole studio went dark the night of the invasion. After the siege was lifted they slowly came back online and started working on a project with my studio. Found out after a few years of talking that apparently Ukrainian SF were posted on the roof and official policy was wipe everything in the event of war, no matter how not-sensitive it was.

Ain't got nothing but good things to say about the Ukrainian team. They're always pushing 100% all day.

[-]

Jestar342@reddit

I (remotely, in London) led a team of devs that were in Lviv, most of whom were volunteers in the reserve army. On the morning of 22nd February 2022 I awoke to find a message from my team:

Russia has invaded, we have been called up to fight. Wish us luck.

I've not heard much from them since, and I have since changed jobs.

Slava Ukraini! Herioam slava!

[-]

compute_fail_24@reddit

Hope they gave em hell

[-]

SpaceGerbil@reddit

Same. One day we had pretty good lead engineers, the next day they were gone and enlisted to fight

[-]

nomadicgecko22@reddit

factories also keep relocating periodically to deal with bombing runs

[-]

TaryTarp@reddit

Watched driving by and see my Director of Engineering get handcuffed by California Highway Patrol.

Was on a bridge, no emergency lane, he got stopped for cell phone use and his driver's license expired. Cop couldn't let him drive and took him to the end of bridge for pickup and could not put him in the police car without cuffs.

[-]

pas43@reddit

This threads makes me appreciate my current work place.

I truly do appreciate being able to call the business owner (front end dev) and lead programmer "fucking idiots" and to tell them their decisions are "dumb", "are you stupid" or plain ole "that's fucking stupid".

I'm incredibly underpaid if I include USA, but since I'm UK based, I'm still underpaid.

But if there pay cut makes me express myself freely and them listen is a truly lengthy thought provoking exercise!

[-]

Leddite@reddit

We accidentally sent 900 million euros to another bank. Luckily they sent it back

[-]

reddit_man_6969@reddit

I would have immediately earmarked 400 mil for legal expenses then strap in and tell em you’re keeping it

[-]

mcmaster-99@reddit

PR disaster though.

[-]

robby_arctor@reddit

That's what the 400 million is for - dealing with the PR disaster of stealing the 400 million!

[-]

pseudo_babbler@reddit

I have heard of others doing this in banking. Banks seem to just deal with it though. Maybe not 900 million euros, but one 60 million AUD error came up recently.

[-]

mcmaster-99@reddit

To a bank or a bank account?

[-]

gopster@reddit

LMAO

[-]

squeeemeister@reddit

I’ve been doing this for almost 20 years now, working mostly for startups and smaller 10 year old companies that think they are startups. I’ve been laid off twice, in the recent years, one pandemic related, the other a small company freaked out at their first negative quarter an fired everyone in the building not under a contract.

I’ve survived 14 other lay off events of varying sizes and approaches. Some have been real cluster fucks, some had angry engineers storm away on their electric scooter never to be seen again, and most just had shocked people with dead eyed stares coming to grips with losing their only source of income.

I think I have some form of PTSD from all of this. I get concerned whenever an “all hands” is scheduled suddenly, or rescheduled, or canceled.

[-]

NoSmarter@reddit

I was on a team where we had a great CTO. He was an older veteran of programming -- a real expert. The whole team loved working with this guy. He then finds a better job elsewhere (not surprising) and they replace him with a complete fucking moron. His only management experience is what he learned from his recent degree from Devry (no, I'm not kidding).

He sends an email to the whole developement team with the following two important directives -- first orders of business:

1) Developers aren't allowed headphones during working hours
2) Upon leaving in the evening, all desk items put away in the drawer, and the desks wiped clean.

He then sent a spreadsheet with names of developers, tasks assigned, and the deadline to do them (which he set). Some of the deadlines were completely out of whack.

He managed to lose the entire team in less than two months. I left after 2 weeks.

[-]

cleatusvandamme@reddit

As a dude with terrible Misophonia, I would probably resign on the spot if management came up with a no headphones rule.

[-]

SoCalChrisW@reddit

Around 2000, the place I worked at had a really small and pretty close IT team. We liked playing jokes on each other, particularly on the network admin.

Back in the day, Exchange didn't validate the from email address; you could easily spoof the from address and the exchange server would happily relay your message looking like it came from whoever. On the Friday before a 3 day weekend, the network admin kept talking about how he hoped they'd let everyone leave early that day for the long weekend. So around 2pm, we sent him an email that looked like it came from the executive team, and looked like it went to the entire company, but it only went to him. This worked great, until he replied all to the email thanking them for letting everyone go early then ran out the door before anyone could stop him. Unfortunately, the way exchange was configured, the reply all did actually go to the entire company, and most of the people assumed they didn't see the original message and left early for the day as well before the administrative assistant saw the email and sent out another email telling everyone that the original email wasn't real, and everyone was expected to work a full day. We got in trouble for that one lol.

Another time I wrote a small program that would quietly run in the background, and at a random time in the afternoon it would slowly move his mouse cursor across the screen for several seconds, and then exit the program so there wasn't a trace of it. We put it in the login script for the network admin so it would run on his computer every day. He didn't say anything, and we quickly forgot about it.

About a year later, the admin was pissed, and ranting as he was packing up his desk. We asked him what he was doing, and he replied that his mouse was fucked up. Apparently, this had been driving him insane. He'd gone so far as to reinstall Windows on his machine multiple times (The login script would copy it from a network share to his local machine, so it persisted across installations), had purchased a new mouse, swapped to a different computer, and tried all sorts of different drivers but his mouse kept moving randomly. He was finally convinced that there was something electrical in the wall that was messing with his mouse, so he was completely moving his desk to try and stop it. When we heard this story, we just sat there silently for a few moments, then all just burst out laughing. When we explained to him what was going on, he was initially pretty pissed, but quickly saw the humor in it.

[-]

Pleasant-Engine6816@reddit

https://thedailywtf.com/

[-]

mailed@reddit

literally my daily read since I was a graduate nearly two decades ago

anything to do with MUMPS is incredible

[-]

jumnhy@reddit

That MUMPS still exists and is so deeply embedded in the health and finance sectors is WILD.

[-]

seven_seacat@reddit

that's still being updated? wow that's memories

[-]

InspectionFrosty4968@reddit

I seriously deleted without where clause on prod and the moment query finished there was a rush of blood to my brain 🧠 after seeing number of records deleted

[-]

Ghjnut@reddit

Had a company advertise it's all company trip to Mexico. Someone took notice that no one would be around and DDOSed the service. Resulted in the whole ops team crammed into a sweaty room at a Mexican hotel for a few hours to hard-line into their terrible internet connection to try to remediate.

[-]

nazbot@reddit

A company I worked for, before I joined, lost their production database with no backups not once but twice.

They also only had their rolling backup set to 7 days, and despite me screaming to change it didn’t and ended up with a bit that lost data which they couldn’t revert.

The lead backend guy got married and left for a honeymoon. He was the only one who had the keys to access the services. He didn’t trust anyone else to have the keys I begged him to give me them so if something happened while he was gone I could at least access the servers. He refused. I got it in writing that he refused. Sure enough there was an outage while he was gone. He was PISSED that I phoned him on his honeymoon and forced him to send us the keys so we could reboot the services that had died.

This is how I learned some basic devops as an iOS developer.

[-]

NeatBeluga@reddit

Why wouldn’t you go to his manager instead of getting it in writing? This is full blown bus-factor

[-]

nazbot@reddit

He was the manager. It was a small team of about 5 people.

[-]

NeatBeluga@reddit

But nobody above him to escalate the potential risk to?

[-]

siqniz@reddit

For me it's always been defending my self against the over critical H1B's that once an app is semi stable look to fire you so they can hire a friend. That was a constant struggle

[-]

kirkegaarr@reddit

A friend of mine was consulting for a startup that was having some tough times. They had laid a bunch of people off and fired the management team, and he was brought in by the board to see what was salvageable and if it was worth investing more in the company to turn it around. They were churning revenue badly, but still had a book of business that was worth millions a year from some very large companies that you've definitely heard of. They ended up making my friend the CEO and he brought me in to lead things from the tech side.

The history of this company was something else. This was not the first time the board had fired the management team. In fact, they had also fired the founder a few years before that because they believed the founder was spending too much money.

The founder also had this group of Russian programmers he knew somehow that did all of the coding. When they fired the founder, the Russians left with him, and the new team had no idea how anything worked, so they decided to rebuild everything from scratch. The problem was that they never really released the new version, and also neglected to support the old version that their customers were stuck on. Hence the revenue churn.

One day we get a serious outage on the legacy system. Upon further investigation, our database is out of space. That's weird. Somehow a table that was full of base64 encoded image files is replicating itself. Also very weird. We keep pulling on the thread and we find that there's an endpoint that will allow arbitrary SQL execution, and it was put there on purpose to build custom reports for customers. Further investigation shows that someone from Russia had connected to this endpoint and executed a script to repeatedly fill up that table with something like `INSERT INTO documents SELECT * FROM documents;` until it crashed the database.

We also got reports from customers that the founder had contacted them about his new competing product he just started.

So basically, board of directors fired the founder. Founder starts new company and retaliates by breaking his old product, and then tried to steal the customers during the outage he caused. At that point I was really starting to wonder just wtf I got myself into. And it didn't end up well.

[-]

trembling_leaf_267@reddit

Worked at a place that made transceivers, the little hardware bits that turn light into electrons and back. We had just switched away from lead solder to tin.

Got reports of failures, and things getting progressively worse, in Texas during a critical season. Analysis showed, in the humidity of Texas, tin whiskers growing out from the new solder, and slowly blocking the light path.

Solution was: whack the containing box really hard. This broke off the whiskers, and restored the communications. Repeat as needed. We did replace the modules over time.

[-]

FelixStrauch@reddit

if (true) // bypass password check

Made it into production for a client app of a major pension fund company.

[-]

DargeBaVarder@reddit

Not a dev story, but closely related.

Was working at a start up. The CTO was friends with someone who was renting a space in a data center, where we hosted basically everything. We rented a rack from him because it was cheap.

All was fine, until one day the friend calls the CTO up and tells him we need to move all our blades. When we asked him how long we had, he told us 12 hours. Yeah, something happened where we had to move a full rack of blade servers to some other rack within 12 hours or all our services would go down.

We rented a rack across the hallway, about 100 feet away, then got 150 foot power and cat5 cables.

We’d run the power cable over and plug it into one of the two PSUs on the blade (and hope the primary wasn’t bad when we unplugged the normal backup).

We then took the Ethernet cable and hot swapped the main cable, because of course we didn’t have redundant NICs. We’d literally pop the main cable out and then pop the new one in while pinging the server and running active service checks. If it went down for more than a fraction of a second we’d swap network cables back.

Once that was done we unracked the still running blade, put it on one of those carts for tools (the grey ones with a big pocket for tools… you know the one), wheel it over to the new rack, and repeat the process in reverse.

We finished the job with hours to spare. Yeah, an entire rack in half a day with no notice. That was a fun day.

Another time we hired a network guy who was great. He set up a whole network modernization plan, then put it into action. The night of the plan he goes up with a few other people. They bring some beer to sip on while they’re doing the work. He has a beer… but didn’t tell anyone he was a recovering alcoholic. He gets plastered drunk in no time flat, and just… never finishes the job. We had to figure it all out the next day.

[-]

adambkaplan@reddit

My first job - my company sold a resource planning/management system for freight companies. It was written well before web services and REST were a thing. Even predates SOAP if y’all remember that.

To turn what would have been a single Windows app into a client/server architecture, they invented a custom protocol where the client sent pipe-delimited strings straight over TCP, receiving similar pipe-delimited strings in response. For testing, we had our own client app that let us send the raw string data back and forth.

[-]

feeling_luckier@reddit

This sounds like something that happened. Is it war story? Not really. It's BAU.

[-]

abeuscher@reddit

In the late nineties I worked at a company that managed imaging for Mass General Hospital. At Christmas one year, one of the engineers put up ~ 250MB of Xmas music on a server. The link to the binary went viral, and the number of downloads crashed the entire Harvard Helix network. We received a talking to. He did not lose his job because everyone involved thought it was funny and, critically, the network came back up very fast and no one was hurt or even terribly inconvenienced. That was the first time I heard the word "viral" in conjunction with the web.

[-]

roosyn@reddit

The biggest lessons I've learned are: - relationships and trust are paramount. Being curious and humble, and respectfully taking people on journeys creates influence. - there are no silver bullets.

On to the scars!

At the start of a project earlier in my career, the dev estimate of 6 months was guided (project managed) down to 3 months. That turned into 12-16 hour days, sleeping under desks, night shifts. At the worst of it, the deployment team swapped to the night shift while training was happening. Total smoke & mirrors. While unpleasant, the team did bond very well and the project went on to live 10+ years before being decommissioned.

A distributed monolith. 10 "microservices" using 18 odd NuGet packages shared across 5 teams. The NuGet packages had interdependencies, meaning a core change required all 18 packages and all 10 services to be released. Branching was ... interesting. Releases usually took a full day to prepare. They all talked to the same database 🤷

A project where the client had been billed for "phase 2" of the project but phase 1 hadn't been finished. The monies for phase 1 had been spent on cars.

Politicians publicly announcing projects and delivery dates before anyone had been consulted - we called that announcement-driven development.

Multiple times where someone's said, "let's create a dynamic form builder!" for a line-of-business app. Always turns into a significant problem for data warehousing and reporting.

Most recently - "can we do that with AI?"

[-]

gopster@reddit

Just a few days ago : A contractor of ours uploaded sensitive information on the app we built onto her personal website. No mocking of data, no blurring of images...like literally everything.
Went through the financial crisis of 2008. Wild times.
Worked for a creep. He would hire women only based on how hot they sounded on a resume. Was obsessed with one of our female coworkers. He would stalk her and harass her online.
When I worked as a QA automation engineer, I had to come into office at 7am to read the results of a mainframe report. This was in 2007. If I didn't make it at 7am, I would get a talking to. Not very automation is it.
Worked for a company that onboarded to Vista Enterprise.

[-]

feeling_luckier@reddit

Stop working for feral losers.

[-]

brazzy42@reddit

Worked on a Java app that was processing XML files representing medical examinations. A typical file was about 1 KB. They could be batched, but could at maximum contain the examinations for one quarter in one doctor's office for one specific disease. We rarely saw files bigger than 100 KB, so processing bigger files was not something we paid any attention to.

One day, the app log showed a crash with an OutOfMemoryError. The file in question was over 13 MB, containing 15,000 cases. I had access to the doctors' registry and found it came from an office that had five doctors. Arithmetically, that would have required each of those doctors to do over 37 examinations for one disease every day and nothing else - not really plausible (and this was pre-COVID).

I wondered: was it a technical error in the office software that replicated data multiple times? Or maybe even fraud, since doctors were compensated for providing these documentations?

Then I called the domain experts, and they had the explanation: it was not a doctors' office treating patients, but a pathology lab, and their "examinations" where histological tissue analyses which could be done en masse, making the numbers plausible.

So in the end, we had to fix our app to process the files in a streaming manner instead of keeping all data for the entire file in memory.

[-]

csanon212@reddit

How long ago was that? I feel like I've read 13 MB files into memory without streaming with no memory problems.

[-]

brazzy42@reddit

About 5 years ago. Note that the file wasn't just read into memory, it was parsed into a DOM structure, which takes a lot more memory, easily 10 times as much. And the app was running as a container with relatively tight memory limits, because there wasn't really any reason it should need a lot of memory.

[-]

cynicalrockstar@reddit

Can confirm, XML serialization was simultaneously the best and worst thing. I made exactly this mistake back in the day also.

[-]

csanon212@reddit

Oh damn, I didn't think it would render onto a UI all at once like that. I was thinking just a backend processing app.

[-]

brazzy42@reddit

It was a backend processing app. DOMs don't just exist in browsers. It's a generic term for standardized ways to represent structured documents as object graphs.

[-]

wlkwih2@reddit

My boss gave a dev conference talk and linked the demo JS with OpenAI key embedded in the source.

We lost $40k that month.

[-]

yellowmonkeyzx93@reddit

These war stories are fun reads! Following this post.

[-]

One-Program6244@reddit

I used to work for a mobile phone company many years ago on the software side. Mostly on UI and features level.

We had a project manager for a new product we were working on who at the last minute wanted to stick a flashing LED into the design of the phone. He thought I can just make it flash by just adding a couple of lines of code. It's simple he said. I told him there was a lot more to it than just two lines of code as there was no drivers written to drive the LED, no hardware abstraction layer and no code in the OS to control an LED. He didn't believe me so I had to spend an afternoon writing a twelve page document explaining how LEDs worked. Ramp-up and ramp-down voltages to control LED illumination, all the other areas of the software stack that will require updating before I can even get to the point of just turning it on in my software.

He relented in the end. This was the point in my career that managers don't always know what they're talking about and are not always right and it's ok to push back.

[-]

DigmonsDrill@reddit

One manager wanted us to integrate with Google Earth. We gave estimates of the time and he said "it's just an app. Install it."

I forget what I said in response, but he never again brought it up.

[-]

briercan@reddit

Early on, I used to work on an emailer app (basically spam...) and I didn't realize that if you left the replyTo email blank when sending an email, the reply would just be sent to the sender, which was a listServe containing all of the email recipients.

Email was sent
Reply "Unsubscribe!"
"Unsubscribe!" was sent to everyone on the list
Reply "Why am I getting all of these emails?!?!"
"Why am I getting all of these emails?!?!" sent to everyone on the list

[-]

trembling_leaf_267@reddit

Wrote half a company's worth of software in LabVIEW: shipping, testing, data handling, database interface layers, business and manufacturing analysis. It was surprisingly fine.

[-]

Historical_Ad4384@reddit

I (M32) once gave a topless remote project demonstration to all the stakeholders. I didn't realize me camera was on. One of my team members did ping me in the middle of the demonstration but it was too late to react and I was already in the last week of my notice. I just continued normally and on one reacted towards the end.

[-]

bigorangemachine@reddit

This maybe my favourite story of creative problem solving.

I had to build this html form that works on this custom platform that is 100% offline. The client had given us an excel sheet with a bunch of weird formula's. They also couldn't give us the formula's 'flat' because no one wanted to pull it out of excel including the PMs. Plus the time was really high.. plus they can change the formula at some point and it'll be hard to update

My solution was basically just to rebuild excel in javascript. I referred to the cells the same way and if a cell updated again the calculations would run again. I also recreated the formula functions.

TBH I think it was easier than anyone sitting down and writing down the whole formula. The formula likely would never change.... I'm sure the HTML is still the same to this day.

[-]

csmithku2013@reddit

All I can really say is there’s a brand of time servers that display sensitive information via a little display on them. Normally the screen is off, but it wakes up if you press a button, oh and background checks on electricians is important.

[-]

RandyHoward@reddit

My story doesn't have much to do with tech itself, just a crazy place I worked...

I worked at this one job for two months about 15 years ago. It was a small ad agency in town and I was hired to build and maintain websites for their clients. On my first day I went out back for a smoke break with one of the guys. He told me to take a good look around at the ground. I saw dozens of used condoms lying on the ground. He said that they're from the owner of the company meeting up with men and having sex with them there. I was dumbfounded because the owner's wife and son also worked there too. A couple weeks later the owner asked me to take a look at why his email wasn't working on his computer. I saw all kinds of emails in his inbox from men trying to arrange a meetup with him. Which I found hilarious because I eventually figured out that his wife monitored every email in the company - there were times I'd email a client and she'd come running into my office with something to say about the content of the email. No idea why she stayed with him, he was not a good looking man and he did not seem to be rich considering the electric company literally showed up to shut off the electric because he wasn't paying the bill.

I also watched this same man fire a guy, and steal his phone before he left. The guy became irate when he couldn't find his phone and started trying to get into a fist fight with the owner. The cops were called and escorted him off the property. Still don't know if he ever got his phone back. I found a new job about a week later, I hope I never work anywhere that crazy again.

[-]

Eligriv@reddit

Back when i was in the final year of my CS school, the school tried a new thing : a partnership with a big company. On one side, the company could make us work for free on a project of their choice, on the other, we would have "exposure".

The company was one of the major telecom companies, and they had a lot of ideas for the "telecom" and "industry" major, for the "media" major etc. But i was in the "AI" major before it was cool. It was a dead end major at the time (like 5% went on and worked in AI or Data). And the telecom guys had no use for us.

So they gave us gigabytes of log files telling us "do something with this". In these logs where clear, plain text, sms (text messages).

That's when i learned that nothing was private and anybody will have access to it.

[-]

Lower_Fish4403@reddit

My first job out of college was with a dot-com that was backed by a very wealthy person who was active in politics. They invested in a small consulting firm's Java framework that ran on New Atlanta's ServletExec. Things ran fine for a while, then one day, the server crashed. For a week, without fail, twice a day the server would start spawning threads and crash. I did everything possible, including working with their experts, but every day at 12PM and 3PM, it crashed. You see, the web application was hosting multiple community sites around political activism and outreach. It was free to sign up and host a community, so anyone could join. Probably no surprise, but even in the late 90's, political websites were a magnet for some crazy people. It turns out this guy was writing massive amounts of content about some conspiracy regarding Christina McGlade and the Nickelodeon show You Can't Do That on Television. When I say massive, his community had almost 10x more data than all of the other communities combined. And it was some creepy stuff. Turns out, this guy realized he could get all his content in by opening like 50 browser windows, entering all of his content in the form, and clicking submit. Threads would back up and the server would spin out of control. Finally, we had to just ban the guy because he wouldn't stop. Unfortunately, he was also our biggest user so the dot-com ended up folding months later.

[-]

JimDabell@reddit

You might like this thread: Events missing in the timeline of web development? There’s a huge amount of things that I haven’t thought about in years in there.

One of my first jobs as a junior developer was working for a small digital agency back at the turn of the millennium. Back then SSL certificates were expensive and PCI didn’t exist. We were pumping out a lot of e-commerce sites, so what we were doing was having a single https://secure.example.com for our company, and taking every card payment from every client site through that, followed by redirecting back to the client’s website to show the order confirmation. A client complained that they could change ?orderId=123 to different numbers and see other people’s order confirmations, including billing information. The lead developer was working his notice period at the time, so the “fix” he dictated was to change the redirect back to the customer’s site to be a POST instead, which would hide the order ID number from the URL. I had to implement this fix across dozens of sites while the owner of the business told clients it was down to “troublemakers”. Even back then as a junior dev I knew it was bullshit, but I had no authority to fix it properly.

Same business, same year: One of our developers would go to a sporting event every few weekends to update their static website with photos from that weekend’s event. This involved putting them up in a hotel and paying overtime. I spent 20 minutes during a lunch break to automate it so that the customer could just drag the new photos in themselves. I got yelled at by the owner because we were charging the customer large sums of money for these updates and wouldn’t be able to justify that if it didn’t take the whole weekend to do. The end result was that the developer would still go to the sporting events, still get paid overtime, but only pretend to work in front of the client while he actually worked on his portfolio. Then at the end of the weekend, he’d drag the files in and go home.

I have a lot of stories like that. Small digital agencies 15–25 years ago were normally complete shitshows where even senior developers often didn’t understand basic things.

[-]

rudiXOR@reddit

Automated bidding for advertisements in an online auction was using a service that predicted the best price for a given auction. On a friday deployment, someone messed something up and it was deployed but due some error used a fallback to the default bid. Nobody noticed that.

So over the whole weekend the company used that value for almost every ad auction. The default value was 1$ ;) think about a large ad network and the company lost millions.

[-]

deZbrownT@reddit

Thank goodness I have nothing to add in this thread.

[-]

NoSmarter@reddit

You must be new

[-]

deZbrownT@reddit

Nah, I just do irrelevant things. 😁

[-]

GistfulThinking@reddit

Our DCs used to operate as WSUS update points.

I entered the wrong IP range into a config, and 50 computers started getting their updates from a remote site.

This was in the early days of windows XP when the internet connections were 1.5mb down and 512k up on a good day.

In my defense, we had a batch of servers that were getting old with defective HDDs, that was my 3rd server rebuild that week, and I autopiloted the other sites IP into the configs.

[-]

HoratioWobble@reddit

I was once brought in to a business as their mobile lead to build an app.

Honestly would take me a month or two alone to build natively per platform OR total if I used React native.

They were a Citrix only outfit and every remote developer machine ran like garbage. Plus we had to build for iOS so I requested a Mac.

It took six weeks to convince them that I needed a Mac to build the app for iOS, then when I got it - it took about 2 weeks for them to agree to give me admin access to install development tools I would need.

I was then made aware that as well as hiring me and my entire team - they had hired a Low code provider to build their app too!

Not just used their service, contracted them to build it WITH their service.

We were then tasked with validating their work? Progress was slow, over the course of 8 months they produced about 8 screens, all buggy in their own low code tool.

I started digging deeper and noticed there were glaring security issues with the produced bundle - and considering this was for a £8bn finance company that wasn't acceptable and I started raising concerns.

No one listened to me, so I e-mailed the CEO directly and in fairness to him he arranged a meeting with me. We discussed the concerns, he then took that and engaged the leaders to actually start addressing this stuff.

We ended up in a meeting, about 30 of us and the offshore low code solution provider who flew over where I was grilling them on my technical and security concerns, they basically said "yeh and we're not changing it" and that was that.

I left. They never released the app.

Another team was using the same platform to build a customer messaging service - took them 2 years to build a simple interface that you had to manually refresh to receive messages.

There's a lot of other nonsense in there but that's the bulk of it.

[-]

Independent_Arm_608@reddit

Thought I was slick by using MSW to mock my API’s so I can build the front end quickly without needing to talk to the backend team

Everything looked good, deployed the app and after a week our SendGrid provider email got hacked and spammed thousands of scam emails.

Turns out the MSW mocks that I used as fixtures were actual production payloads and it got sent to the front end as part of the JS bundle.

[-]

0Iceman228@reddit

Was forced to make an unsecure calculation to prevent the collision of two large stacker cranes driving on the same rails. They refused to add proper safety hardware for exactly that job. Well I subtracted something which should have been added in the formula. Next morning I got a call telling me they crashed. At least nobody was dumb enough to blame me cause I would have just driven home and never showed up on that site again. I was the lead (safety) developer.

They added more safety hardware after that. Though it was still a stupid design.

Had a building automation project where I had to manipulate the database entries during lunch break so I could show everything was fine when we went back to testing. The servers crashed during lunch because of a software bug we couldn't fix ourselves.

I never want to go back to PLC automation.

[-]

shooteshute@reddit

Was at a startup that ended up stealing our pension contributions and not paying people. Some people still haven't even been paid for 3+ months of work

It's mad because you could see the company was poorly run. We had very few customers, a product that didn't solve anybodys problem and genuinely the worst senior management I've ever worked under.

It's crazy how optimistic people remain, even right up until the end. Thankfully I jumped ship after our 3rd payday in a row was overdue