Dependency Hell: The Hidden Costs of Dependency Bloat in Software Development

[-]

TheOtherZech@reddit

Part of me wonders how much of dependency hell comes from the fact that version pinning and vendoring tends to be handled at the project level. Monorepos kinda sorta help with that, but monorepos aren't a universal solution. Some sort of abstract hierarchy of workspaces, where each workspace can pin/publish/vendor resources, could work, but that's a lot of infrastructure and not the kind of infrastructure that easily scales down.

Ends up being one of those things where it feels like it could be easier, if we "just" changed all of our tools and all of our workflows and built our data centers on lay lines and made deals with the aes sídhe to make all of the intractable parts magically go away.

[-]

SnooSnooper@reddit

Monorepos in my experience have a different but similar problem: once they are large and worked on by many teams, instead of the slowdown coming from having to update a dependency version in many places, you have to coordinate all the project teams to update code consuming the dependency, and naturally they all have different priorities and schedules, so this is very hard to do.

Source: I worked on a monorepo, and there was a dependency that was something like 20 major versions behind, because we couldn't coordinate all the product teams to update their code to latest.

[-]

valarauca14@reddit

Some sort of abstract hierarchy of workspaces, where each workspace can pin/publish/vendor resources, could work, but that's a lot of infrastructure and not the kind of infrastructure that easily scales down.

git submodule remains the greatest feature nobody actually uses.

[-]

TankAway7756@reddit

Almost all dependency hell could be avoided with the simple rule of not breaking shit in the code.

Just the other day at my job I had to deal with a library that used to consume a well known, but alas still concrete class in its public API.

Consuming an interface rather than said class certainly makes for better design, but breaking perfectly working and secure code by removing the first pathway altogether rather than deprecating it is madness.

[-]

LeapOfMonkey@reddit

https://xkcd.com/1172

[-]

Big_Combination9890@reddit

"Hidden Costs"?

https://en.wikipedia.org/wiki/Npm_left-pad_incident

As a result, thousands of software projects that used left-pad as a dependency, including the Babel transcompiler and the React web framework, were unable to be built or installed. This caused widespread disruption, as technology corporations small and large, including Facebook, PayPal, Netflix, and Spotify, used left-pad in their software products.

If that's "hidden", I guess the asteroid that wiped out the dinosaurs would have been "hard to notice".

[-]

teslas_love_pigeon@reddit

My unpopular opinion was that the owner had every right to do this, please remember that npm took away his name of another popular library because some shitty startup wanted the name for themselves.

[-]

throwaway490215@reddit

My dependency on a DNS resolution & access to github & docker are totally different.

Bro. Trust me bro.

[-]

BeautifulCuriousLiar@reddit

fucking corporations

[-]

that_leaflet@reddit

This article says a lot about how dependencies may have vulnerabilities so you should avoid dependencies, but ignores the fact you still need some functionality the dependencies provide.

So instead you have to implement the functionality yourself. And of course, your implementation may be buggy and insecure. At least with those dependencies, you have so many more people using them, discovering bugs.

[-]

that_leaflet@reddit

This article says a lot about how dependencies may have vulnerabilities so you should avoid dependencies, but ignores the fact you still need some functionality the dependencies provide.

So instead you have to implement the functionality yourself. And of course, your implementation may be buggy and insecure. At least with those dependencies, you have so many more people using them, discovering bugs.

[-]

ThisIsMyCouchAccount@reddit

I'm not sure I really understand where "dependency hell" comes from.

While I'm certainly not new to development I have only done things for the web. And sure, back in the day towards the start of my career we didn't have all the tools and frameworks.

But is a framework dependency hell?

Or are we talking about adding a bunch of random libraries?

[-]

International_Cell_3@reddit

"Dependency hell" traditionally refers to situations where you have transitive dependencies causing conflict. This could be explicit (you depend on A version 1 and want to use B version 1, but A requires C at version 1 and B requires C at version 2). Some package managers will just fail at this point.

More nefarious are when C is updated with an incompatible change that breaks A or B but it's non trivial to downgrade C or upgrade A or B to handle the breakage.

The even more nefarious situation is when the package manager/language allows multiple versions of the same dependency to be linked into the same program, but doing so causes unspecified behavior because of global state or multiple definitions. This depends on the language and ecosystem and it's a big reason why package managers historically avoid allowing multiple versions of the same dependency.

[-]

ThisIsMyCouchAccount@reddit

How often does this really happen?

I've never experience anything like that.

The closest I've experienced is once on a big project a library we were using for all our API calls was abandoned and we had to switch out to a different one.

[-]

zten@reddit

It happens painfully often.

My main experience is with Java and I’ll frequently find mutual incompatibilities in libraries because of different major version requirements for Google Guava. To deal with this, more popular libraries will use shade/shadow to create a separately namespaced copy of it for their own private usage — but they typically won’t do that if their library depends on Guava data structures in their public API.

[-]

International_Cell_3@reddit

I mean it's enough of a problem you're commenting on a thread with dozens of upvotes about it. It's still really gnarly when you hit react dependency issues in JS, especially if you upgrade dependencies religiously.

But it used to be worse. Modern package managers use automated constraint solving algorithms (meaning package A doesn't depend on B, it depends on B=^maj.min.patch where ^ means "compatible with" (or any of the other common constraints used by semver). There are some interesting things about those algorithms, but the gist is we automate the solving today whereas 10-15 years ago you manually solved your versions and often patched things when they didn't work.

Imo I think the rarity of these problems is actually a downside because the solvers/package managers aren't guaranteed to do an optimal job (or at least, better than a programmer) at managing dependencies (they often have constraints not expressed by the packages), and when they fail, it can be in spectacular ways that requires deep understanding of not only their packages but for example, the way modules are loaded by javascript runtimes (or whatever linkage model your language has).

[-]

ThisIsMyCouchAccount@reddit

Maybe that's what it is.

It's not an every day problem but when it is a problem it's a real problem.

The people talk it seemed like it would be very unlikely that I had never encountered it and I haven't. So I thought maybe I was missing something.

Maybe it's as simple as that I've spent a huge majority of my career doing client work. It's unlikely that a big issue would happen in the random 6 - 18 months I would be one a project.

[-]

max123246@reddit

The reason it's a problem is that a lot of people are working on teams with 20-50 people, many of whom they've never talked to, all trying to create something with their own slightly different mental model of the project as a whole

That's how you end up in a situation with circular dependencies. Someone adds a dependency here, another person adds one here, and over a couple years you'll eventually hit the issue

[-]

ArticleWaste8897@reddit

It also depends quite a bit on your tech stack, especially if you’re dealing with legacy code. Amazon has version sets and dynamically links thousands of dependencies into huge Java projects. One forced upgrade (like Java version) can lead to spending weeks unfucking your version set with 10000 things in it.

Google builds nearly everything from source at head -> the only possible source of conflicts is 3p libraries which they use certain strategies to minimize conflicts with.

[-]

compchief@reddit

If you rely more on a framework rather than independent dependencies, you will have much less if any problems since the people working the framework handles the packaging for you and there are often widely discussed standard "goo-to" dependencies for things the framework does not handle.

Strictly talking about somewhat popular, used and maintained dependencies.

[-]

FlyingRhenquest@reddit

It happens in all the languages and with shared libraries a lot. I had a project where I was building packages for three separate servers that were supposed to be identical but always seemed to have different shared library versions from server to server. I ended up just throwing my hands up and building my code statically just to avoid the constant failures on one system or another.

[-]

FlyingRhenquest@reddit

I had that happen with Ruby once, on a project where a dependency had a library upgrade that required a newer version of the language, while another dependency had gone unmaintained and needed the older version of the language. Of course, as a C++ programmer, I need to consider every dependency I add very carefully since CMake uses global variables and all it takes is one jackass changing one of them in his find-package instrumentation to screw my build up for days while I track down why the build system is ignoring my build flags.

[-]

YahenP@reddit

Well.... let's take pure vanilla wordpress for example. Let's install the basic package u/wordpress/scripts . And.... suddenly 300 MB of JS libraries appear in the node_modules folder.

[-]

ghillisuit95@reddit

Yeah. I think the key is to make sure sure you can keep track of what dependencies are used where, to make sure that dependencies are kept up to date, and to make both of the above auditable easily. This functionality likely needs support from your ci/cd system

[-]

Chii@reddit

its more like you need to have a local repository as part of your CI, rather than depend on an external third party and the internet.

It's a different dependency hell problem than security and verifiability.

I want my CI pipeline to be able to (to be) run in complete isolation.

[-]

InterlinkInterlink@reddit

It inevitably comes down to developer discretion and discipline. Should you rewrite the entire world of software for your application's functionality? In the majority of cases - no. That doesn't make importing the world a good a idea either (let alone necessary).

I am of the opinion that too many developers are overly-permissive with dependencies and are incapable of asking very basic questions to assess dependency risk. It's another vector for technical debt, and the historical career churn of leaving a company/role before shit hits the fan only amplifies the problem.

[-]

Vectorial1024@reddit

I say this is specifically JS's fault. How come no standard library replacement for is-even?

[-]

HolyPommeDeTerre@reddit

x % 2 === 0 ? Isn't that standard ?

[-]

Vectorial1024@reddit

Sigh my sweet summer child...

Consider the following:

// detect an even number
let x = null;
console.log(x % 2 === 0);
// true

Clearly. that's not expected behavior.

is-even may look like a meme, but it is not. It is a genuine production-grade package, and it is worthy of every GitHub star that we can muster.

[-]

Yawaworth001@reddit

That's just a lack of understanding of the language being used.

[-]

International_Cell_3@reddit

You're not type-checking x before using it in an operation that requires x: number, yes JS is weird in how it defines these operations but the fuckup is let x = null and not x % 2 === 0.

[-]

valarauca14@reddit

You're not type-checking x before using it

Damn, when did Javascript gain strong types?

[-]

International_Cell_3@reddit

Even in weak, dynamically typed languages, some programs requires type checking variables:

if (typeof x !== 'number') { throw new Error("x must be a number"); }

In JS in particular this is whenever doing something numerical. You should put this check way up the call chain to avoid doing it in a hot loop. Some (bad) developers rely on unit testing for this.

[-]

Vectorial1024@reddit

... Did you notice I mentioned JS but not TS?

[-]

International_Cell_3@reddit

Yes. You're still fucking up by not typechecking if a variable is a number before using it in a numeric context.

[-]

Signal-Woodpecker691@reddit

On our project we are under strict instructions to avoid external dependencies as much as possible. This eventually included having to systematically remove dependencies on a code library developed by a separate team in the same company as us to do a lack of visibility and documentation of what they were doing.

[-]

Dragdu@reddit

And of course, your implementation may be buggy and insecure.

It might. But by making a custom solution for my needs, I might end up with just 1/10000th of the scale and thus avoids design vulnerabilities. After all, I wouldn't write my custom logger to have the log4j vuln.

All these are just another dimension along which you need to evaluate the trade offs.

[-]

plumarr@reddit

That's why I implement my oidc client myself. /s

[-]

loup-vaillant@reddit

Speaking of dependencies, it would be nice if I didn’t have to activate JavaScript, not just on the main domain, but on tailwind as well. Like, you’re depending on a third party service to render your website, for a blog this is absolutely insane, you’d need a very good justification for this.

I don’t have the patience for this any more.