What are your favorite resources on dealing with software complexity?
Posted by dondraper36@reddit | ExperiencedDevs | View on Reddit | 53 comments
The more I have to solve problems in software engineering, the more I agree with the idea that the main goal of software developer and software architecture is fighting accidental complexity.
This is why I really admire talks, blogs, and books that focus on minimizing complexity and embracing simplicity instead of trying to sell a specific architecture, framework or pattern.
Some of the highlights from me:
- Almost all talks by Rick Hickey, especially Simple Made Easy.
- Out of the tar pit This one is a phenomenal paper on software complexity
- Destroy all software - small videos on different topics. Functional core, imperative shell is a particularly great one even though I have to admit I have yet to learn to apply it.
- A Philosophy of software design - a very pragmatic book with a lot of insights on what makes a system unnecessarily complex
- Grokking simplicity - I am currently reading this one. It's a great primer on functional programming that deliberately uses JavaScript, obviously not language you first think of in the context of FP. But the idea was to show that FP principles can make any code simpler, not just pute FP language based.
What are your favorites?
AffectionateData1252@reddit
Software design patterns are scalable. Take SOLID or any of the gang of four patterns. They apply at the small scale and at the large scale.
Eventually any system is complex, but if it can be analyzed and reasoned about at varying degrees of depth, a programmer should be able to grok it.
Unit tests go a long way toward explaining how the system should work.
pgomez1973@reddit
I like Juval Lowy's notion of volatility-based decomposition. See the book Righting Software.
Synyster328@reddit
OpenAI's
o1-preview
oro1-mini
models.jimjkelly@reddit
All the recommendations here are good. Another thing we should be comfortable with is that we are going to sometimes need to make the wrong system to better understand the right system, and as long as we are evolving to address very concrete lessons vs just falling into the “replace things because we don’t like them” trap, we should accept that we’ll do so rearchitecting form time to time, and occasionally it’ll be painful.
If it’s worth doing you should find it possible to make a solid business case for the value of doing this, especially if you aren’t chicken little about it and can show incremental gains and not be trying to stop all work to make things perfect.
wvenable@reddit
I remember I had to design a very complex compensation system into our software. We had a simple system but our clients were coming to us with much more complex requirements. So I designed and built a new compensation system with all the requirements. But the moment it was finished I knew exactly how to build the right compensation system. I discussed it with the team and I immediately starting building that instead. I actually don't remember if we ever released that intermediate one.
Once everything was built that first time, it was easier for everyone to see how to fit it together in a way that made a lot more logical sense for us and the users.
agumonkey@reddit
so common and true
AmateurHero@reddit
I just did this with a refactor a really complex UI component. The original wasn't written in the best way (mostly due to time constraints), so I wanted to explore a few different ways to express the concept.
I ended up with 3 different stashes over the course of an afternoon. All of them were viable. There was one that heavily leaned into class hierarchies and inheritance, and on paper, it felt like it was the more natural way to express the concept. Yet it was very orthogonal to how everything else was designed.
The 2nd build out felt organic within the context of the repository. I was initially leaning to the 1st build out, but after coming back the next day, I thought that 2nd one might be a better choice for the team. I asked one of our juniors to take 5 minutes look over the stashes. They were much more comfortable with the 2nd stash.
syklemil@reddit
Yeah, Fred Brooks' "plan to throw one away" remains relevant. We're going to need exploratory phases, and we're also going to need to throw most of the stuff from that phase away.
Well, that and Alan Perlis'
hippydipster@reddit
I think it goes well beyond planning to throw one away. The ideal, IMO, is to always be reconstructing parts of the system and never stop. It should always be a part of maintaining and building upon an application.
One of the problems this approach reveals is when we build systems too intertwined to do this. The lesson we should learn is, don't let our systems become too intertwined. But, unfortunately, the lesson we seem to learn most often is "don't fix what ain't broke", and then one day it is broke and we've gone so far down the path of spaghettification that we're screwed.
syklemil@reddit
I partially agree and I think all of us here also know that preventative maintenance is generally a good thing, but I also think there's a difference between building a prototype, exploring various approaches, getting a bunch of (important!) negative results; and that kind of preventative maintenance.
This ties into the importance of getting science to publish negative results: With "how do we X?" as the question, it's good to know that approaches a, b, c, e and f don't actually work, and approach d has some horrible shortcomings. The WD-1 through WD-39 that we don't actually use were important paving stones for getting to WD-40.
Of course, time doesn't stand still, so what might've been a working solution in the past might no longer be, and vice versa. But that's different from the kind of stuff you see that works in staging but has some horrible flaw exposed in production where your only real option is to discard the solution, because the world as it exists turned out to be different from how we thought it was.
But yeah, there are lots of techniques to make that process less painful, including good code structure, automated builds (which a surprising amount of places apparently still don't do), automated tests (same), feature flags, canary deploys, blameless postmortems, etc, etc. These are all ultimately tools for doing the wrong thing in the least painful manner, because we are going to keep doing the wrong thing in our search for the right thing.
Rain-And-Coffee@reddit
Sleeping honestly.
Can’t mention how many times i have solved or better understood complex systems by just sleeping on it.
You can only cram in so much knowledge in a day before you get diminishing returns.
Also a cup of coffee and some blocked off time helps wonders for really making sense of things.
agumonkey@reddit
that's something I learned late, and it's more general than software.
your body and mind have limits, stress them up next to those, then rest
stacking more hours blindly might only work for young students and I'm not even sure they get that much benefits beside the dopamine hit (meaning the clarity and depth of understanding might not be there).
same goes for music, or sport.. rest is part of the growth
troxy@reddit
and shower driven development, and dog walking development, and drive home development.
I definitely have text message chains where I have thought of something and sent it to my coworkers to tell them to remind me the next work day.
kex@reddit
Eventually the problem becomes "how do I turn off my thinking so I can sleep"
foomojive@reddit
I thought of something while watching a movie with my wife the other day, I was like STAAAHP
The_0bserver@reddit
Take a walk development.. I remember being stuck on a problem for a few days, and then I just took a walk. Started solving that problem right there, and the walk went on for 3 hours. Took that long to sort through my thoughts. Came back. Tried it. It worked.
eightslipsandagully@reddit
Don't forget shitting driven development, occasionally a toilet visit causes inspiration to strike
tcpukl@reddit
I lost count of the number of bugs I've fixed in my sleep. Get to work in the morning, type up the fix in 5 mins and it works! After you spent the day confused and getting nowhere, or so I thought.
syklemil@reddit
Yeah, it's similar to rubber ducking. I'd also include taking a shower or a bath, maybe getting out to touch grass and just stare at nature in solitude for a while, or just staring out the window for a while.
Letting your mind unwind and wander is important. As any athlete knows, rest is important, as is having a training programme that isn't just full intensity all the time.
erehon@reddit
Also, defuse mode thinking if you get stuck while focusing. I have a lot of random insights about a problem I’m solving while showering or taking a walk thinking about something else. Slacker driven development
Maxion@reddit
As a parent of some toddlers, I fully endorse this message for president.
wvenable@reddit
I've put things on the shelf for a while because I had difficulty with design only to come back to it a month later and suddenly it felt dead simple to do. Obvious, in fact.
Chevaboogaloo@reddit
I always have my breakthroughs when I can't stop thinking about a problem after I clock out for the day.
bigorangemachine@reddit
Ya I agree.
There only thing you should do pushing hours is writing, manual testing or mindless copy-paste tasks that you can't automate.
frontenac_brontenac@reddit
The best resources I've encountered were not talks or books, but practical challenges that taught me a deep, almost tactile sense of software complexity.
The first was practicing typed functional programming in the ML tradition. Standard ML, OCaml, F#. If you've never programmed in such a language, it's hard to overstate just how simple they are. (I am not talking about Scala and Haskell, those are a lot of people's first contact with typed functional programming and they can be fiendishly complex.)
When you code in a language that is simple, there is no ambient complexity clouding your thoughts and obscuring the differences between different solutions. Simplicity becomes a matter of skill, rather than personal taste.
Typed functional programming should be enough to "jailbreak" the average working programmer. It's worked for me, it's worked for my peers, it's worked for my pupils. When they backport their understanding of the fundamentals into a mainstream programming language, they can code far beyond their years of experience. The converse is also true: senior software developers with a career's worth of experience in Java or C# frequently fail to write code that "simply works", all development is done by groping in the dark and iteratively approximating the spec until either it becomes "good enough" or the runway ends.
For those who've tasted simplicity and would like to go yet further: Software Foundations is an interactive textbook, almost a video game, about the formal verification of programs. You write code, and then you prove properties about your code. Proving doesn't happen automatically, you need to come up with arguments as to why your code actually works and encode those arguments into a proof script that the compiler then verifies.
As you get better at proving that a piece code abides by its specification, you begin to get a sense of which algorithms, and which branches of which algorithms, are going to prove vexing. Any code whose correctness isn't immediately, trivially obvious becomes a cause for alarm.
At this point you should begin to notice the insanity that is mainstream programming practice in most of the industry. Programmers treat their own code like a black box, they hammer it left and right until it slots into the shape it's supposed to occupy, at which point everyone understands that it is not to be touched or disturbed in anyway lest it totally and fractally dislocate. This isn't every team in the industry, but it is most teams at second- and third-tier companies, and even a surprising number at FAANG-tier companies. People bitch that the tech interview doesn't resemble day-to-day practice, and they are correct, but if you can't infer a loop invariant from a problem statement, then I'm sorry but your code only ever works by accident. Go back to basics.
In exchange for this existential despair, you will be able to effortlessly perform feats of clarity and correctness in software engineering that will leave your colleagues open-mouthed and slightly concerned for their jobs. And the process of getting there isn't awful either.
hippydipster@reddit
I remember when I ran into such a programmer who had this effect on me. It was truly eye-opening. I'd been a professional programmer for 10 years at that point and had accomplished a lot, but I realized the difference in clarity between the code I tended to write and what he wrote. Was just astonishing.
codeconscious@reddit
Yeah, as a primarily C# engineer, I would say picking up F# casually as my first functional language has helped me become a better developer to some degree and has helped me look at writing software from a different angle. (I'm still early in my F# journey, admittedly.)
edgmnt_net@reddit
Yeah, I can definitely trace at least some of my abilities to stuff like Haskell and the surrounding ecosystem (communities, articles, blogs etc.), even if I did not use it as a primary language in an official capacity (or, ok, I might have done once but it was entirely my call). And more generally, exposure to multiple paradigms (and OOP coming rather second).
Another on my part was exposure to open source. The good large projects are very strict and very no nonsense about stuff. In many ways, a completely different world compared to typical company projects (which can be quite awful at times) and you don't have to land that really good job to get the good experience. Enterprise stuff tends to live in its own bubble, with its own lingo and oddities; it may be a very large bubble, but it's still a bubble.
Another was being a relatively generalist dev. I have had enough exposure to be able to dive in and pick most stuff up. It's easier to make connections if you know a bunch of stuff. You may be able to pick stuff you work on, get involved in other areas, unblock things, avoid boredom and make a good impression.
If we're talking about less experienced devs, I doubt a hands-on interview is going to be to their advantage. My impression is that leetcode or generic theory is a compromise in that sense. If the interviewer brings in the actual project code on a laptop, will they be able to demonstrate anything meaningful in, say, one hour? Can people handle actual large codebases as a transferable skill? That's also one thing that requires experience to learn and generalize.
old_man_snowflake@reddit
I have followed this mantra -- almost to a fault. Complexity kills software. Every time there's a new framework or technique, it's always billed as "X but simpler!" Simplicity and maintainability are the two most important technical traits of a project, IMO. Sure, making money is the raison d'etre, but from a technical success standpoint, the most long lived projects aggressively fight complexity.
The test for me is how long it would take to hand off to a competent development team. And if it's not "read this doc" why not?
hippydipster@reddit
Another way to put it: Flow considered harmful. People in flow write a lot of code that they probably shouldn't have. Flow is like System 1 thinking. Very fast. Very productive. Prone to logical errors.
Cut the flow and reflect on what you're doing more often.
Obsidian743@reddit
There are two problem discussing simplicity: no one agrees what simplicity means and no one talks about what the increased complexity solved (they just talk about the problems it creates).
It's simpler to new() every object up as you need it. But no one disagree that DI is a better approach.
It's simpler to write a bunch of
if
statements instead of using interfaces or patterns like factory.A monolith is much simpler than a distributed system but we know what kind of problems you run into with that.
Deploying by hand is technically much simpler than automating everything.
Is SQL is simpler than NoSQl? Redis simpler than Kafka? Are VMs simpler than Docker and Kubernetes?
Everyone's experience is going to be different. The fact that I've seen lots of mistakes one way or the other pan out in the long run means I'm likely to make different decisions than someone who hasn't. But that means if the complexity causes problems, people only see the problems. They don't see all the problems we didn't have.
bwainfweeze@reddit
Refactoring by Martin Fowler.
The difficulty in woodworking is not in designing the table. It’s in having the skill to build the table that’s already in your head.
GoTheFuckToBed@reddit
I record the amount of time an energy required to work or understand X. Then decide what to do with X.
e.g. Interns get repeatedly stuck on docker and everything docker brought in. We removed docker from their flow.
dondraper36@reddit (OP)
Oh, yeah, I should have also added Grug brained dev and https://boringtechnology.club/
Maxion@reddit
Grug now put club down.
brsmith080@reddit
I was totally looking for grug brained dev
Pleasant-Database970@reddit
+1 for Destroy All Software: Fcis, extract value objects, separate business logic from application logic.
DreadSocialistOrwell@reddit
Books and sites on Design Patterns have saved me many times.
It comes with more experience to know which one to use and also I have a habit of relentlessly refactoring. It doesn't have to be exclusive to the entire system or even to limit you to a single one.
Also, if you're in the OOO world, knowing when to use Interfaces vs Abstractions (proper abstractions) and not just because reasons.
Weekly_Victory1166@reddit
Pad of paper and a mechanical pencil, access to source code.
DigThatData@reddit
cannabis
gilmore606@reddit
grugbrain.dev
lordlod@reddit
Book: The design of everyday things.
Once you are experienced I feel it is more exposure to technical talks and conferences. Learning how others have tackled issues and what other systems are around. When you realize that this problem looks like that thing you saw the presentation on two years ago, and can grab a library that does half the job for you.
TeamHelloWorld@reddit
Hire good engineers who know how to ask for help & when to ask. Add conventions to the code ASAP too.
Sleep is the other one, but someone beat me to it.
roger_ducky@reddit
My own understanding is to make every single function look like pseudo code or configuration file (ie “declarative”/ functional definitions)
New programmers and AI tends to slowly grow a single function until it exceeds the limit of understanding. My own thing about that is: Excluding boilerplate, goal is to keep stuff as succinct as possible. Preferably below 10 major blocks of code. (Loops, if blocks, etc)
wwww4all@reddit
The main goal is make money. Make money first, then improve things as needed.
abe_mussa@reddit
Echo what others are saying - mainly through experience
Earlier on in my career I really struggled to apply knowledge from videos, books, blog posts etc. it felt like every time I tried to do something to reduce complexity, I’d end up making it worse
Worst experience I ever had was with CI tools that would give warnings for cyclomatic / cognitive complexity
But years later I don’t often consciously think about it. After years groaning at coming back to overcomplicated, difficult to understand code (which often git blame informs me is my own) and learning which pitfalls to avoid, you get a good sense for what is acceptable complexity
Obviously the resources are important, you need the concepts. But would advise against actively trying to reorganise code to reduce complexity if you’re working with just those resources alone - practical experience is 99% of it
engineered_academic@reddit
Sensory deprivation tanks. Seriously.
CpnStumpy@reddit
Favorite resource: experience
The more years I'm at this, the more I see the cycle repeating, the more I see the same problems and same patterns emerging. After enough years fighting the battle for simplicity, plying the skill becomes increasingly easier.
Read, study, think, write. Teaching is perhaps the best way I've found to learn - and I mean in the trenches, not academically.
YMMV
Tons of other good resources of course. But as a lazy programmer, experience would be my favorite because short of TBI I always have it
Familiar-Flow7602@reddit
Domain Driven Design - tackling complexity in the hearth of software
wedgelordantilles@reddit
Grug brain dev
i_exaggerated@reddit
Just say no.
Cercle@reddit
For a practical implementation, some languages allow you to set linting rules on computational complexity. Yes, they are very simplified measures, but at least provide a starting point. Thanks for the reading list !
ivoryavoidance@reddit
youtube.com/@ants_are_everywhere