CVE reduction worked until the next scan. Is rebuilding on someone else's patch schedule a strategy?

Posted by Any_Artichoke7750@reddit | linux | View on Reddit | 4 comments

Six months of the same cycle. Critical CVE drops, we rebuild, scanner clears, three weeks later another one surfaces from a transitive dependency we didn't even know was in the base image.

The runc disclosures in November took 9 days before Alpine had anything clean upstream.

Nine days of sitting on it, giving stakeholders timelines we made up, waiting for someone else to move. No SLA, no ETA.

Tried switching base images twice. First switch broke builds for 2 weeks.

Second got us to distroless which helped with CVE count but snapped 4 services that needed shell access during incidents so we rolled back under pressure. My teammate ran the numbers last quarter. 22 person-hours on rebuild cycles triggered by base image CVEs we had zero control over.

Is anyone off this treadmill or is the answer just that you pick a base image and accept that this is part of the job now.

[-]

Firm-Goose447@reddit

See, CVE reduction doesn’t mean you’re safer, it means you’re temporarily aligned with what’s known today. The next scan doesn’t undo your work, it just reveals how incomplete the model is. Teams that scale don’t aim for zero CVEs. They aim for controlled exposure (what’s exploitable, what’s reachable, what actually matters). Everyone else ends up stuck in this loop of cleaning yesterday’s vulnerabilities while tomorrow’s are already queued.

[-]

IWritePython@reddit

This is basically throwing up your hands. Fine as far as it goes.

First, time to exploit is 1 day in a quarter of cases (28%) 44 days in the average case, and some are exploited on a negative timeline now. (from mandient report if you wanna pull it). So leaving stuff unpatched or waiting for upstreams to patch is not good.

That's not exactly the same as remediating CVEs, but CVE remediation is an indicator. Really you're wanting to go beyond remediation but if you're not remediating then you're probably not tracking upstream either.

What do I mean by upstream. I mean what the maintainer is shipping not what a community pulls in 4 months later. The maintainer is the upstream ultimately. They often hear about disclosures a bit ahead (not always). If you track upstream you're at lesat doing what you can without jettisoning the software, forking it, doing something really weird. And some claudebaby can't just look at your prod infra, look at versions, pull some highs and chain something together to ruin your day.

I'm a Chainguard engineer and at Chainguard we create a rolling distro that tracks upstream very tightly. You're basically running on real upstream. And you're at ~0 CVEs so very little noise.

Nine days of sitting on it, giving stakeholders timelines we made up, waiting for someone else to move. No SLA, no ETA.

This is a very interesting writing cadence you have O.o funny feels like I've been seeing it a lot lately.

Anyway. People are always like it's about what's reachable. That seems like shit. First, not that easy to tell. Second, even if not "reachable" can still affect the env in ways. Third, if it's not fucking"reachable" why the gosh darn it do ou have it in there.

Just do it right, have a minimal image that does what ou need and no more, track upstream. I think we're the way to go but ther'es other ways to get there, it's real work but that's ultimately the goal.