Why auto-fixing secrets in CI doesn’t really work
Posted by WiseDog7958@reddit | Python | View on Reddit | 44 comments
I have been messing around with automatically fixing hardcoded secrets in Python projects. the idea sounded simple,
detect secrets in CI - rewrite them to env vars - done.
Technically it works. you can do safe rewrites with AST and keep it deterministic. but people really don’t like CI modifying their code.
Even when the change is safe, it still feels off. the main things I kept hearing,
- CI should be read-only
- people want to see changes before they happen
- auto-fix in CI feels like losing control
After a while I kind of agreed with that. what seems to work better is splitting it,
- CI --> detection only (fail the build)
- fixing --> done locally (pre-commit or manually)
So CI enforces the rule, but you’re not letting it touch your code.
how are you all handling this?
do you let CI fix stuff, or keep it strictly read-only?
billFoldDog@reddit
Have the CI fail the build with a message.
Have your code linters alarm for secrets.
dogfish182@reddit
CI changing stuff is an off practice in my opinion. CI should just be a set of deterministic failure gates that all can run locally. This gets more important as AI gets more prevalent. It should just say ‘code isn’t good for reason. Fail’ or not.
If youre detecting secrets in CI, great, fail and alert
Smallpaul@reddit
What is the issue with CI making predictable, deterministic and well-documented changes to branches?
I’ve had CI that automerges and generates API documentation which gets committed etc.
Rarely seen any big issues with it. What have you seen?
dogfish182@reddit
Annoying devs mostly. Commits you’re ’not responsible for’ being owned by you is the implication.
WiseDog7958@reddit (OP)
pretty aligned with that.
CI as a read-only failure gate makes sense. once it starts mutating code, you lose determinism and trust.
the part I’m focusing on is just pushing the fix earlier. instead of,
CI detects - fail - dev fixes
handle the obvious cases locally before commit, and let CI stay as a strict verifier. so CI doesn’t change anything, but also doesn’t keep failing on things that could’ve been fixed safely upfront.
dogfish182@reddit
If all your deterministic gates run locally and ci just runs those then the shift to precommit is pretty easy
Effective-Total-2312@reddit
Pre-commit can easily be skipped by developers, but they can't skip CIs (or at least they shouldn't have the permissions to skip them).
Pre-commit is auxiliary only, CI is the real place where all required checks should run before allowing a merge.
dogfish182@reddit
I’m not suggesting replacing ci with precommit, but if both do the same thing then a long feedback loop is then the developers fault and the problem is solved
WiseDog7958@reddit (OP)
it looks easy on paper, but in practice most local gates are still just detection. pre-commit catches things, but it usually stops at fail and fix manually.
the tricky part is safe fixes.
once you try to auto-fix locally, you hit the same problem as CI. some rewrites are fine, others are technically valid but semantically wrong.
that’s where I think the gap is not just running gates locally, but deciding what’s safe to fix vs what should be refused.
yopla@reddit
Let's call it CD then. But having the CI rebase trivial merge conflict it's actually pretty useful.
Especially since most of the dev are just asking Claude to do it anyway.
dogfish182@reddit
I find getting Claude to rebase harder than doing it manually, it’s usually unexpected, it bangs into things and costs a lot of turns and therefore money. I find it personally easier in an ide that lets me left/right compare
Aggravating_Type7759@reddit
yeah we had similar issues at shop where I work. tried auto-fixing linting stuff in CI and developers got really annoyed when their PRs suddenly had commits they didn't make.
now we just fail the build and make people fix locally. takes bit more time but everyone feels better about it. we use pre-commit hooks for most of automatic fixes - catches secrets, formatting, basic linting before it even gets to CI.
only exception is maybe documentation auto-generation from docstrings, but that goes to separate branch anyway so nobody gets surprised by random commits in their feature branch.
Theonetheycallgreat@reddit
For your precommit hooks, are you just using them on a monorepo or is there multiple repositories you've applied it too. Just wondering because we allow developers to make repositories whenever they want and so we have quite a lot of different types of repositories in our organization. Makes things more difficult.
WiseDog7958@reddit (OP)
there is exactly where it gets messy.
In a monorepo it’s easy, one pre-commit config and you’re done.
but In multi-repo setups, it usually turns into shared template repos, org-level pre-commit configs or CI enforcement when local hooks aren’t guaranteed.
but that’s also where things break down in practice, not every repo is consistent, and not every dev has hooks set up properly. that’s kind of why I’m focusing on making the local fix path as frictionless as possible, so even in messy setups the obvious stuff still gets handled before it hits CI.
how you guys handle hook consistency across repos ?? do you enforce it or just rely on CI failures?
Theonetheycallgreat@reddit
For secrets we use github's built in push protection and then for secrets that get past github has an Ai detection. Along with that we have a secret scanner on every pull request set at the org level. I'm sure a few still get by since secrets can take any form and we get alot of false positives from the Ai detection.
WiseDog7958@reddit (OP)
yes that tracks especially the false positives.
once it’s happening at PR level, you’re basically dealing with both noise and stuff slipping through. what I kept seeing was a lot of these were just dumb hardcoded values locally, but they only get caught way later. so even just reducing how many of those reach PR seems to help a bit.
shoffing@reddit
We made autofixing work in CI by having it create a pull request with the original PR's branch as the base branch, sending it to the author for review. Author can review the changes and merge the bot's PR to accept them. Or, author can fix the changes themselves, after which the bot will close its own PR.
We also do the same thing with screenshot diff tests, where authors can review changes to screenshot goldens as a bot PR before accepting them (or fixing a legitimate regression).
WiseDog7958@reddit (OP)
this is exactly the failure mode I’ve seen too. auto-fixing in CI sounds nice until people lose trust in what’s being pushed to their branch. what you described (fail in CI + fix locally with pre-commit) is basically the stable pattern.
The gap I’m trying to close is,
pre-commit catches things, but doesn’t always fix them safely.
so instead of just failing, handle the obvious cases automatically (hardcoded creds - env vars) and refuse the rest. that way devs don’t get surprise commits, but also don’t have to manually clean up the trivial stuff every time.
DrMaxwellEdison@reddit
I use a compromise: I don't want the CI making changes, no, but I want the changes that it could make to be easier.
So, I mixed a CI that runs pre-commit that performs local auto fixes, with reviewdog/action-suggester. Changes aren't committed directly, but are sent as suggested changes back to the PR. Then I can choose to commit those suggestions from GitHub's UI or do them on my machine then push them myself.
I don't do anything beyond deterministic checks at the moment, so that much is enough; and I keep control on the process.
WiseDog7958@reddit (OP)
that is a pretty clean compromise. especially for keeping control over changes. the only part I kept running into with similar setups is the review overhead for really trivial fixes.
like if it’s something clearly safe, even going through a
suggest - review - apply
loop starts to feel like unnecessary friction. that’s kind of where I’ve been experimenting with pushing those cases fully local, and leaving the suggestion/review path only for anything ambiguous. if you’ve run into that, or if the suggestion flow is fast enough that it doesn’t matter.
Glathull@reddit
If CI detects secrets, you’re already fucked and so are your secrets.
There’s no point in trying to “fix” an exposed secret. The only safe option is to fail.
CI should kick it back to you and force you down the secret rotation path.
WiseDog7958@reddit (OP)
once it’s committed, rotation is the only real fix. yes agreed
what I’m focusing on is the earlier case - before it ever becomes a commit. a lot of these are just obvious hardcoded values in local code (quick scripts, internal stuff, copied examples).
so the idea is to catch and clean those up locally before they turn into a rotation problem at all.
sudonem@reddit
I handle this by… not storing secrets in my code in the first place.
It’s trivial to add support for passing secrets as cli arguments or environmental variables or .env files.
If you are running apps with hard coded secrets, your code is bad and you should feel bad.
WiseDog7958@reddit (OP)
In a clean setup, secrets shouldn’t be in code at all.
In practice, they still show up more than people expect - especially in internal tools, quick scripts, or copied examples. most teams rely on scanners after the fact. I’m focused on catching and fixing the obvious cases earlier, before they become a commit history problem.
Yrrsinn@reddit
When a secret left the devs computer via a commit, it is by definition a commit history problem. Invalidate the secret and rewrite the code.
The check should happen on the devs computer, maybe by running a check before pushing
WiseDog7958@reddit (OP)
that’s basically where I landed too. once it leaves the dev machine, you’re already dealing with a bigger problem. so catching (and fixing where possible) before commit seems like the only place it really works cleanly.
2ndBrainAI@reddit
Totally agree on splitting detection vs fixing. CI auto-modifying your code is a trust problem more than a technical one. Even if the rewrite is safe, you lose the audit trail of what changed and why, and people feel like the pipeline is doing things behind their back.
Pre-commit hooks work way better for this. detect-secrets or gitleaks locally blocks secrets before they ever hit the remote. CI then just enforces: fail the build, show a clear message telling devs how to fix it locally.
The vague "secrets detected, build failed" message is the real killer. Teams start bypassing the check just out of frustration, which defeats the whole point.
WiseDog7958@reddit (OP)
yes the “fail with no clear path to fix” part is exactly what breaks these setups.
once it turns into,
detect - fail - dev has to figure it out manually
people either get frustrated or just bypass it.
I’ve been experimenting with pushing more of that into the local stage, handling the obvious cases automatically and refusing anything unclear instead of just failing. still trying to figure out if that actually reduces friction or just adds another layer.
Smallpaul@reddit
People should see changes before they happen is defensible as a position.
“A human being needs to author the commit on local” is not and it’s going to be totally untenable in the next couple of years.
Software should author the diff/commit and the human should just sign off on it. Why would you ever want a human doing work that a script can do?
ThiefMaster@reddit
You can do your AI slop. I'd happily WRITE code instead of reviewing it. Because the latter, for code written by a machine or by a very junior human developer, requires pretty much the same time it would take a senior/experienced developer to write it - except that by writing it yourself, you also know it inside out!
Smallpaul@reddit
You seem to be under this mistaken assumption that we are discussing AI code generation. The tool being discussed is just Python code using the AST module to call Python code.
ThiefMaster@reddit
To me "Software should author the diff/commit and the human should just sign off on it." sounded like you want something to generate code for you (not necessarily related to OP's tool)
Smallpaul@reddit
I was speaking generally. The author’s tool where appropriate and an LLM where appropriate. And yet other tools (whether deterministic or stochastic) as appropriate. The right tool for the job.
WiseDog7958@reddit (OP)
agree with - automation should do the work. the problem is where you trust it. auto-authoring commits in CI sounds great until you hit cases where the rewrite is technically valid but semantically wrong (string building, partial secrets like these). that’s why I’m drawing the line at “provably safe” cases only.
for those - I’m fine with automation doing it locally before it ever hits Git.
everything else - I’d rather refuse than silently mutate code in CI.
so it’s not anti-automation. it’s anti-unsafe automation.
Smallpaul@reddit
If the automation is not trustworthy then a human should confirm it. E.g. merging a child PR.
WiseDog7958@reddit (OP)
cases where you can’t fully trust the change >>>>>
the part I’m trying to avoid is pushing trivial fixes into a PR flow.
if something is clearly safe (like a hardcoded value - env var), adding a PR + review step just adds friction for something nobody actually wants to review.
for anything ambiguous though, I’d agree, better to surface it and let a human decide.
Smallpaul@reddit
You are kind of talking in circles. I said “why not just automate the commit”. You said: “maybe it’s not a safe change. Needs a human decision maker.” So I said “okay, make it a PR.” And you said “but maybe it’s safe and it will be annoying to have to review the PR.” Either you consider the transformation safely automatable or you don’t. I assume that in general you don’t.
But in this particular case there is a lot more value in catching the error before it leaves the local computer so you should use pre-commit.
WiseDog7958@reddit (OP)
I think we’re talking past each other a bit.
I’m not treating it as a judgment call like ,'maybe it’s safe'.
The idea is, only automate cases where the transformation is structurally provable and doesn’t change behavior. if it meets that bar - automate locally.
If it doesn’t - refuse entirely (not PR, not CI rewrite, just surface it).
so it’s not a grey area. it’s a strict boundary between “provably safe” and “don’t touch”.
Haunting-Form4664@reddit
This is one of those ideas that sounds clean in theory and then hits culture head‑on in practice. CI quietly editing your source feels like someone committing to your branch while you’re not looking, even if the AST rewrite is perfectly safe. People lose trust in the pipeline way faster than they gain it.
prassi89@reddit
trufflehog + pre-commits: https://github.com/trufflesecurity/trufflehog . Or even better as a claude hook on git commit.
Shift left
WiseDog7958@reddit (OP)
trufflehog + pre-commit is solid for detection. the part I kept running into is what happens after it flags something.
most setups still end up with,
detect - fail - fix manually
I’ve been experimenting with handling the obvious cases locally (like hardcoded values - env vars) and refusing anything unclear, instead of just failing. ended up building a small CLI around that idea:
https://github.com/VihaanInnovations/autonoma
not sure if it actually fits better than existing workflows, but interesting on how people would want that to behave in practice.
WiseDog7958@reddit (OP)
that “window of exposure” point is fair. but it depends on where the fix actually happens. If the fix is local (pre-commit stage), the secret never reaches the repo or CI in the first place, so there’s no exposure window to close. end-to-end automation in CI only helps after the commit already exists.
So for me it’s more about shifting the fix earlier rather than making CI more aggressive.
omg_drd4_bbq@reddit
why are secrets even being committed? use injection
WiseDog7958@reddit (OP)
In ideal secrets shouldn’t be committed at all. In reality, they still are.
Not because people don’t know better, but, quick scripts, internal tools, copied examples, rushed commits.
Most scanners catch it after the fact. I’m trying to handle the boring case earlier — the obvious hardcoded values that can be safely moved to env vars before they ever leave local.