Experience is what you got when you didn't get what you wanted
Posted by Icy_Screen3576@reddit | ExperiencedDevs | View on Reddit | 79 comments
Good times teach only bad lessons: that building software is easy, and that you don’t need to worry about risk. The most valuable lessons are learned in tough times. In that sense, I’ve been “fortunate” to have lived through some hard ones:
- The .NET Web Forms era, which started as a drag-and-drop success like Windows Forms but fell apart in production with the ViewState mess.
- The Adobe Flex wave, where companies went all in on rich browser apps until Apple pulled the plug on Flash in Safari.
- The run toward NoSQL, where teams rushed to use MongoDB everywhere and ditched relational databases, only to hit a wall when the first serious report was asked.
- Installing ERP systems for users who needed only 5% of what those systems were built for, and watching the learning curve kill morale.
- ORM-heavy code that boosted developer productivity but struggled under real read load.
- The microservices trend, where everything became a service, and we paid the distribution tax.
- Kubernetes setups that were harder than the systems they were supposed to run.
I can't help but wonder how this will look 10 years from now.
arstarsta@reddit
All of what you said is nothing compared to when we have to clean up the mess of AI vibe coding.
Icy_Screen3576@reddit (OP)
Now, one person can create the tech dept of 10 engineers. We will be here to help.
Basting_Rootwalla@reddit
Alas, the real 10x engineer has been found.
Icy_Screen3576@reddit (OP)
Hahahahahha
bmain1345@reddit
And look how we’ve evolved! Modern ORMs (EF Core cough cough) are extremely performant if you know what you’re doing
Kuresov@reddit
The main problem is the ORM being taken for granted and devs not understanding (or even considering) what it’s doing, or doing wrong, under the hood.
They’re convenient until lazy loading is silently broken, or unintentionally creating n+1 queries, etc.
christian_austin85@reddit
My personal fav is my old team not setting limits on any entity, so most every field is nvarchar(max).
caprisunkraftfoods@reddit
I feel this so much. I'm a big advocate for Django because it solves so many more complexity problems than it creates, the ORM performance issue is #1 line I hear in opposition. On the rare occasion you get them to share an actual example it's nearly always a profound lack of competence with relational databases in general, the ORM is immaterial.
BusinessWatercrees58@reddit
Can confirm from personal experience. But also from personal experience, taking these into consideration after being bit in the ass from not doing so still makes ORMs pretty great to use, once you know the pitfalls to consider.
raven_785@reddit
My issue is that every attempt I've seen to avoid using an ORM eventually resulted in a shitty custom built ad hoc ORM, even if it wasn't being called that. I've fallen into that trap myself even knowing better.
Thegoodlife93@reddit
What do you mean ad hoc ORM? I just use Dapper to execute raw SQL queries that are mapped to model classes. If you have a strong SQL background than the SQL is much is easier to read and write for complex queries than ORM queries and you can tell exactly what's happening (at least up to the level of the execution plan. I'm assuming there is a way to see and modify that in Dapper but I've never tried).
ThatSituation9908@reddit
In some ways, but as long as you have control over the underlying code that defines the query and avoid generic sql template helpers, then I think you’re fine.
I have done ORM-less data models in DDD. It’s fine. My pain point of this is it is easy to create a lot similar data models without a centralized ORM: e.g., the application data model can differ from the table schema.
Icy_Screen3576@reddit (OP)
No matter how far i tried, moving to dapper was always a better solution.
bmain1345@reddit
Just didn’t know what you were doing ;) jk yea dapper is way more performant but the gap is being closed!
Icy_Screen3576@reddit (OP)
Just recently,,, i had a worker service consuming broker messages and saving to an oracle flat table. The pod utilized 4-5gb memory and 6-8 cores with entity framework. Moving to dapper reduce to normal 150k-200kb.
bmain1345@reddit
That’s quite literally insane, I’m skeptical of too many entities being loaded into change tracking that don’t need to be, usually that’s a huge performance whiff. But you already moved and it sounds like it’s working great so props 💪🏻
Icy_Screen3576@reddit (OP)
Nope. Its a single flat entity.
WaterIll4397@reddit
The nosql one never made sense to me..human decision makers think in terms of spreadsheets.
ericmutta@reddit
Excel is the OG DB ✌️
caprisunkraftfoods@reddit
The point was aspirational performance concerns. You can't linearly scale a relational database across multiple machines the way you can an application server. So it was a mix of "ok but what if we went viral", and the first generation of devs who'd never touched physical server hardware graduating to senior positions.
Heavy-Report9931@reddit
I've come to the conclusion that even in python lamd where sqlalchemy is shoved in your face all the time. its just easier to just learn sql and write the damn queries yourself lol
nickjvandyke@reddit
Have you seen SQLDelight? You write plain SQL queries, then it generates types and methods accordingly. Feels so good. I wish I could use it everywhere (it originated for Android, idk if it's expanded yet)
Heavy-Report9931@reddit
Will take a look at this. I've only ever used Room for android.
nickjvandyke@reddit
Ah Room! I remember that being a distinct improvement too. But iirc I moved off it to SQLDelight.
ThatSituation9908@reddit
A good chapter in what ORM-less looks like in Python. I love it because you are free to use all the neat typing features that SQLAlchemy did not have but now likely caught up.
https://www.cosmicpython.com/book/chapter_02_repository.html#_the_normal_orm_way_model_depends_on_orm
superide@reddit
That's how I did it in simpler (most) jobs. When you're forced to go from a limited list of stored procs you might forget
Icy_Screen3576@reddit (OP)
You mean going back to the basics.
Junior_Platypus2102@reddit
this framing helped me a lot after a failed promo cycle last year. i didnt get the title but i got extremely clear feedback about what was missing, which was all communication and influence stuff not technical skill. that clarity was worth more than the title would have been because i actually knew what to work on instead of guessing
BackgroundWash5885@reddit
The "distribution tax" is definitely the big one of this decade. It’s wild how we consistently find more complex ways to solve problems that "boring" tech already handled perfectly fine.
rupayanc@reddit
The NoSQL rush is the one that aged the worst for me. Watched teams confidently rip out Postgres for MongoDB because "web scale," then spend the next two years rebuilding their reporting layer from scratch because the first real business query broke everything. The top comment about React/SSR going full circle is another good one. What nobody's saying out loud yet is that agentic AI might follow the same arc. The ceiling hits when the first audit shows up and the answer to "what did this agent decide, and why" is basically a shrug.
Icy_Screen3576@reddit (OP)
Two points taken from comments thus far: the react/ssr, and the servicestack going commercial. Your audit concern of ai agents is intriguing. I guess we have to wait and see.
raven_785@reddit
I know I'm nitpicking here, but you shouldn't be "asking" reports of your OLTP database.
MongoDB falls apart when your data turns out to be actually relational, which many systems evolve into even when MongoDB was fine initially. Or when you really needed transactions (at least when MongoDB was most hyped).
Icy_Screen3576@reddit (OP)
Good one. I have rarely seen this separation between operational and analytical data. I worked with an owner who ask for prototype/mvp from us, then let his paid customers use it for a while, later to ask for detailed report the day he is meeting them. The teams who stayed with traditional sql/oracle had no issues delivering. Unlike l, teams who followed that hype and struggled to deliver the same one time.
CerBerUs-9@reddit
I have yet to see something work well so I must be getting so much freakin' experience
Icy_Screen3576@reddit (OP)
Good luck!
Deranged40@reddit
Good judgement comes from experience. And experience? Well a whole lot of that comes from poor judgement...
AdUnlucky9870@reddit
Lived through most of these. The one I'd add is the "rewrite it from scratch" wave that hits every org about 18 months after a successful v1. We did this twice — once justified, once catastrophic. The catastrophic one taught me more about software engineering than any book.
The pattern I see across all your examples is the same: a technology gets adopted for what it enables, not for what it costs. The cost only becomes visible under load, at scale, or when the original team leaves. NoSQL was fantastic until someone needed a JOIN. Microservices were elegant until the network became the bottleneck. K8s was powerful until the team maintaining it outnumbered the team building the product.
The current one that worries me is the AI-assisted coding wave. Not because it's bad — it genuinely accelerates certain work — but because we're building a generation of developers who can produce code faster than they can understand it. That gap between production speed and comprehension speed is where the next "hard lesson" is hiding.
AdUnlucky9870@reddit
great list. I'd add the GraphQL-everything phase — where teams replaced perfectly fine REST endpoints with a GraphQL layer that nobody on the frontend actually needed the flexibility of, but now you have a resolver graph to debug instead of a straightforward controller.
the pattern across all of these is the same: new technology solves a real problem for the people who built it at massive scale, gets evangelized to everyone else as a universal solution, and then teams spend 2-3 years learning why the old boring thing was actually fine for their use case.
the one that taught me the most was the microservices migration. we broke a monolith into \~30 services because that's what you were "supposed to do" circa 2017. took us 18 months to get back to the same reliability we had before. the distributed tracing alone cost more engineering hours than the original app. eventually consolidated back to 6 services at boundaries that actually made sense.
my rule now: you earn the right to add complexity. if you can't articulate the specific scaling or organizational problem a technology solves for YOUR team, you're just adding resume-driven architecture.
AdUnlucky9870@reddit
the k8s one hits different. we spent 6 months migrating a monolith to k8s because "everyone's doing it" and our infra complexity went up maybe 10x while our actual reliability barely moved. the old monolith on 3 VMs with a load balancer was honestly fine for our scale.
biggest lesson I took from living through the microservices wave: the distribution tax is real and it compounds. every network hop is a failure mode. every service boundary is a contract negotiation. we went from "deploy one thing" to "coordinate 14 deploys and hope the integration tests catch the contract break."
what I'd add to your list: the current AI coding wave. not saying it's bad — we use it daily and it's genuinely productive. but I've already seen teams ship stuff they don't understand because the model wrote it and "it works." that's the .NET Web Forms pattern all over again. works until it doesn't, and when it breaks nobody knows why.
the pattern across every one of these is the same: new tech solves a real problem, industry overcorrects, applies it everywhere regardless of fit, pays the complexity tax for 3-5 years, then settles into the boring appropriate middle ground. every single time.
AdUnlucky9870@reddit
The ORM one hits close to home. We had a Django app that was beautiful to develop against — clean models, readable queries, devs were shipping fast. Then we hit real traffic and realized half our endpoints were doing N+1s nobody noticed because the ORM made it so easy to traverse relationships. Took weeks to unfuck.
The pattern I keep seeing is: the thing that makes adoption easy is usually the thing that hides the footgun. ORMs hide SQL. K8s hides infra. Microservices hide coupling (until they don't). And now AI is hiding the fact that people don't understand the code they're shipping.
10 years from now we'll be on this list talking about the vibe coding era the same way we talk about the NoSQL rush.
AdUnlucky9870@reddit
This list is painfully accurate. I'd add one more: the GraphQL-everything phase. We went all in on it for internal service-to-service communication because "one query language to rule them all." Turns out when both sides of the wire are your own teams, REST with good contracts is simpler to operate, debug, and trace. GraphQL shines at the edge for client flexibility — using it everywhere was solving a problem we didn't have while creating ones we did.
The pattern I keep seeing is that each wave starts with a real insight (NoSQL IS better for some workloads, microservices DO help at certain scales) but then gets cargo-culted into contexts where it doesn't apply. The industry optimizes for "what's the hot new thing" instead of "what's the right tool for this specific problem at this specific scale."
Ten years from now, I suspect we'll look back at the current AI-assisted development wave the same way. The insight is real — these tools genuinely accelerate certain kinds of work. But the "just vibe code everything and let the model figure it out" crowd is going to hit the same wall every other wave has hit: the hard problems haven't changed, we've just found new ways to defer them.
AdUnlucky9870@reddit
I'd add the current AI-assisted development wave to that list. We're in the "this changes everything" phase right now. In 5 years we'll have a much more nuanced understanding of where AI coding tools genuinely help (boilerplate, test generation, exploration) vs where they silently introduce debt that compounds.
The pattern is always the same: new tool appears → productivity spike → "why would we ever do it the old way" → edge cases accumulate → painful correction → mature understanding. The teams that come out ahead are the ones who adopted early but stayed skeptical.
Your Kubernetes point especially resonates. We spent 18 months migrating to k8s for a service that ran perfectly fine on two VMs behind a load balancer. The "distributed systems tax" is real and most teams don't account for it until they're already paying it.
TylerDurdenFan@reddit
I think there's space in there for a point about abandoning server side rendered pages to fully embrace Facebook's React, complex state on browser, to eventually a decade later rediscover SSR as a "cool thing"
raven_785@reddit
The SSR that I see being pushed these days only makes the complexity worse. It's not replacing React with backend templates and JavaScript sprinkles, but it's an extension of React that for the same views sometimes renders them server side and sometimes client side. And from what I've seen it's a shitshow.
Izkata@reddit
Or both at once, with "hydration".
Business-Row-478@reddit
Eh that had more to do with compute costs and internet speeds changing over time than what does / doesn’t work
TylerDurdenFan@reddit
I think there were also many other factors back then.
Just to stir up controversy: The early adoption of React had a bit of a cargo cult factor (companies imitating Facebook and Instagram hoping to make it big), and also a bit of Resume - Driven Development (probably wise for those starting their careers back then).
I think that it might be today's EPYC compute power and fast speeds what is making SSR (and even monoliths) attractive again, with DHH preaching how bootcamp never ever needed to shard, and OpenAI serving 800 million users with a single Postgres database (although with a lot of read replicas)
Franks2000inchTV@reddit
I think a lot of things that made react special have been adopted elsewhere now, so it's hard to remember how cool it was when it first came out.
funguyshroom@reddit
If all you worked with before React was jQuery it might've felt like the second coming.
Also JSX was a big innovation. MVVM frameworks before that trying to mesh with html were kinda wonky.
stayoungodancing@reddit
State updates can definitely be caused by both of those, though, alongside just figuring out how to manage it asynchronously. It’s definitely useful and necessary at times, but it’s not going to absolve the issues faced by slow SSR responses, either.
Franks2000inchTV@reddit
I wonder if we could use SSR to send a bunch of Javascript down to the browser and just have it render the whole app.
stayoungodancing@reddit
I’d love for other frameworks to start overtaking React at this point, or if there was less of a reliance on immediacy via state. I feel like it’s led to the lesson that just because you can, doesn’t mean you always should.
Icy_Screen3576@reddit (OP)
Definitely i missed that one. The spa framework everywhere syndrome, if we can call it that way.
Gunny2862@reddit
Title sounds like an inspirational quote.
xaervagon@reddit
Yeah, the web world took off like a rocket and today it looks like a wild west of tech and paradigms. It would be cool to see something like POSIX for the web world where things get standardized and developers can train to it. It would solve a lot of problems (like HR trying to hire 10 years exp in flavor of the week solution).
I did Visual C++ longer than I care to admit, so I always felt like an outsider looking in to the modern tech world. In general, it looks like most tech comes and goes with a massive hype cycle and either finds its niche or fades away entirely when VC can't get their ROI. I honestly dislike the hype/marketeering driven development of the modern software world, but there's nothing I can do personally about it; there was something to be said for working with a well defined platform/tool/whatever with defined purpose and full fledged tooling and documentation.
Icy_Screen3576@reddit (OP)
One mental model that helped me: Trend Is Not Your Friend.
xaervagon@reddit
No argument from me on that one. Whenever I encountered another tech hype cycle starting up, one of the things I would ask myself on the job is: what does this do for me? Does any of it solve my problems?
The other thing about the tech hype cycles is that it isn't really being sold to us, the software developers. Today, it's being sold to the management. If you work at a small company or deal with a salesman manager, the fomo is real with these people.
Icy_Screen3576@reddit (OP)
The fomo is real to us as well. People choose architecture to enhance their resume or to experiment.
superide@reddit
Resume driven development is too real when you've been laid off and realized a lot of the tech world has pass you up.
Companies individually update their tech in a stair step pattern. But taken together, the "average" movement starts to look like a gradual slope, which is why it can feel like the industry is always moving. You then have to hope where you work doesn't stand on the same step for too long
xaervagon@reddit
I know. This is how modern tech motivates people to teach themselves on their own time instead of actually paying for training. This reduces costs and shifts burden back on to the individual instead of the company. One of the biggest causes of LinkedIn lunatics used to be software devs trying to keep up with trends.
raulmonteblanco@reddit
We go back to linux / apache / mysql / perl. Just kidding, but good point.
superide@reddit
We need to make the first letter of a stack acronym L for Linux again
Icy_Screen3576@reddit (OP)
And .cgi and shared hosting 😂
raulmonteblanco@reddit
I'm working on a project now that uses blazor, and I'm worried that it will be the next Web Forms.
thekwoka@reddit
Depending on the ORM in question, this may not always be that bad.
Cause you can get the flexibility and clarity in dev, and in actual bottlenecks optimize the queries simply to bypass the ORM query driver stuff.
Icy_Screen3576@reddit (OP)
My experience with entity framework was bad. I now use dapper for read and nhibernate for write.
xampl9@reddit
EF is best when there’s a 1:1 mapping between table layouts and data models.
Which only happened to me in the most trivial of cases.
I like to point out that a lot of MSFT technologies are for looking great at conferences. “… and we’ve got a complete solution in only 20 minutes!”
Uhh no.
trojans10@reddit
Yea. When should you use or not use an orm?
thekwoka@reddit
When the ORM causes query cascades.
The best ORM are mostly just query builders.
bluetrust@reddit
Yeah... I'm like, activerecord is fine. Drizzle/prisma is fine. I can't remember any of these making poorly performing queries under load. Poorly performing queries under load is from devs who don't put fucking indexes on their queries, or do something dumb with their database schema.
thekwoka@reddit
Yeah mostly.
But things like Django's ORM can be really shitty since they can result in query cascades really easily, and until.somewhat recently it was all synchronous....
rocketonmybarge@reddit
I agree with everything except the ORM heavy code. I built a 100% ORM access app with Servicestack's Ormlite, and have encountered no bottlenecks. I think the underlying problem is that Entity Framework allowed devs with little to no sql experience to "query" the data, cause severe performance problems due to lack of understanding of how EF works.
Icy_Screen3576@reddit (OP)
You made me recall the servicestack lesson i should have added. Relying fully on a framework that suddenly goes commercial. That was a painful experience.
rocketonmybarge@reddit
I mean, Demis had to pay for developing it, and it was an absolutely bargain to use compared to WCF. We never upgraded to the paid version till years later because we didn't have to.
chills716@reddit
Being an architect, people are shocked when I say the experience gained isn’t from seeing good or great systems, it’s all the ones that I’ve seen that didn’t work. My mantra is, I know what’s works because I’ve seen what doesn’t.
DigmonsDrill@reddit
I have a long list of things I thought were a good idea that weren't.
SpiritedEclair@reddit
Negative space architecting.
joexner@reddit
But ORM's aren't bad, usually, and microservices and k8s are there to help with big teams being hard. They're kinda-ok answers to hard problems. Monoliths full of obscure raw SQL don't scale.
Saki-Sun@reddit
I'll pay the rest of your points.