How do you answer 'is this commit in prod?' in a mostly-autonomous pipeline?
Posted by Square_Voice_1970@reddit | ExperiencedDevs | View on Reddit | 13 comments
Our pipeline is roughly 90% autonomous now. Last week an AI agent patched a critical CVE, merged it, and shipped to prod. Our security lead found out five days later. Neither of us had a clean way to answer the question "is this fix in prod yet?"
The doing is automated. The telling is not. Right now, answering that basic question means stitching together CI logs, registry tags, k8s manifests, and Slack - basically AI-as-Sherlock, hoping it connects the right dots.
What patterns are people actually using for deterministic "is this fix already in prod?" queries across an autonomous pipeline? SBOM + provenance attestations? A homegrown deployment ledger? A Backstage plugin? Curious what's working in production, not what vendors claim works.
engineered_academic@reddit
Has nobody ever heard of a service catalog these days?
WildWinkWeb@reddit
Best solution for now is aggregating everything into a single catalog (deploy events/runtime data). I know Port, our IDP, can do this. Unsure if there’s an easy Backstage fix.
Wide-Pop6050@reddit
What? How do you not know what's in prod? You don't know what the final, running version of your code is?
mechkbfan@reddit
How does your CI/CD tool not tell you what commits are getting deployed?
We're using Shitbucket and it tells you
You can use git version or similar too if need to
If I see any product mentioned here I'm going to talk shit about it
davvblack@reddit
you seriously let AI merge and deploy random shit?
SirIrrelevantBear@reddit
Your cd pipeline can create an annotation on your deployment or a configmap with the commit sha if the merge. Same for any other runtime. Your telemetry should also ingest that as an attribute to your logs and metrics so you can distinguish between versions and catch regressions.
Empanatacion@reddit
I don't follow. You at least know that service X is running v1.2.3.4 of your code, yah? And you've tagged git? Where's the disconnect?
R2_SWE2@reddit
Am I the only one cynical enough to think OP has an alt that’ll come along shortly to pitch a product he has developed?
At any rate, if your org has no QA to check what’s autonomously being pushed to prod, then who cares? YOLO, right? Sounds like your org is well on the way to making the news in a bad way.
Expert-Reaction-7472@reddit
nope... happens here several times a day.
spez_eats_nazi_ass@reddit
what kind of fucking moron let's clankers do that?
edgmnt_net@reddit
If the commit can only be on one branch, that's easy, it's practically
git merge-base --is-ancestor. If it can be on multiple branches (although not technically correct since they're different commits) then it becomes much harder to tell in the general case, since stuff may require adaptations. It's easier to just keep a log of what gets merged where.EmberQuill@reddit
How was the CVE found? Whatever told you it was there should also be able to tell you that it's gone.
Morazma@reddit
When you say "is it in prod" are you talking about the gap between merging to master and deploying prod?
Don't you have a history of pipelines where you can see merges being deployed?