Approved an AI feature for production knowing the security review didn't actually answer the question. It shipped, nothing happened, I still don't feel right about it.
Posted by johnypita@reddit | ExperiencedDevs | View on Reddit | 51 comments
The feature was an internal tool that routes support tickets to the right team using an LLM.
The security review went around for a week
Three groups signed off. The form had checkboxes for vendor approved, data classification, retention reviewed. All three got checked and I was the engineer of record so I got the final yes.
I sat on it for two days. The thing the form did not have a checkbox for was the thing I actually cared about
the ticket bodies sometimes contain customer credentials that customers paste in by accident. We strip the obvious ones but we dont catch all of them. Nobody can tell me what percentage we miss, because nobody has measured it, because measuring it would require sampling production tickets, which is its own approval cycle.
so I asked. The cisos office told me the existing controls were "appropriate to the risk class" when I asked what the risk class was I was told it was "internal tool with vendor under DPA" which was technically true and answered nothing
I approved it. The form got signed. The feature shipped. Six weeks later its still running and there has been no incident Im aware of
I gave a yes that I knew was answering a different question than the one I was actually being asked. The form I signed says the feature was reviewed for data risk. What I actually signed off on was that the review process had been completed. Those are different things and the form doesnt know the difference.
I described my own decision in the audit log as "approved per security review, low residual risk." The "per security review" part is doing all the work. I am pretty sure that is how every senior engineer in every company is currently approving every AI feature, and I know that is fine but still it feels off
Anyone else been the last signature on a chain that everyone is treating as "the review" without anyone having actually done the review? How do you live with it?
dorkyitguy@reddit
Congrats! You just used AI to do what your ticketing system could probably already do, but for a much higher price!
engineered_academic@reddit
Security reviews at most companies aren't serious. They are enough to say they followed a process so that their cybersecurity insurance will cover them. Once you understand this, a lot of pressure is off.
Ok-Leopard-9917@reddit
I mean maybe at your company but lots of places take security very seriously.
engineered_academic@reddit
No company takes security seriously. It's a cost center for the company, to be minimized as much as possible. Tech companies are the worst of the bunch. Hell even security tech companies like Aquasec and SolarWinds have had massive massive security breaches that could have easily been prevented with just a few simple additions to their security plan.
If companies took security seriously no developer would want to work for them because they would find it too "restrictive".
Trust me I have worked at places where if the SOC got a vulnerability report they would shut your shit down before you could even get your ass out of bed.
Ok-Leopard-9917@reddit
Some companies do take security seriously. Though large companies vary widely by group so it depends on the product.
nkondratyk93@reddit
nah, the form failed you not the other way around. if that data type had no checkbox, close the gap - retroactive audit + update the checklist. staying guilty does not patch anything.
lionelpx@reddit
Can your customers send emails to your company - are these emails hosted by a third party (ms365 / gmail) - how is it different ?
johnypita@reddit (OP)
its diffrent cause private cutomer credentials doesnt get to an ai model in an email
lionelpx@reddit
I meant more - what’s the concern ? Data leak or appropriation from a third party seems at similar risk level in both cases (llm / email), and in both cases mitigated by the same contractual clauses and provisions
false_tautology@reddit
If you're using Outlook 365 and Copilot, its actually likely that they are.
johnypita@reddit (OP)
we are not using co pilot exactly because of these reasons
false_tautology@reddit
I'm impressed with that. Seems like everybody is enabling Copilot in their emails.
johnypita@reddit (OP)
yeah
i was actually surprised about that in the begining as every place i worked in before this place enabled it,
but turns out this all filed of llms and privacy is still kind of grey area, and our company dont trust it at all
FinanceSenior9771@reddit
yeah this is the classic failure mode with security reviews that are process-driven instead of evidence-driven. the fact that “it shipped and nothing happened” is basically luck, not control.
what i see work in real orgs is: (1) define the actual threat in plain terms (ex: llm sees or logs customer secrets in ticket bodies), (2) require a checkbox or artifact that proves the control addresses that threat (ex: tested redaction coverage with a sampling plan), and (3) add monitoring with a clear abort condition before you scale the feature.
for your specific gap, the concrete ask is measurable redaction + handling: capture a representative sample of prod tickets (or use a synthetic mix validated by your security team), run them through your stripper, and report miss rate + what happens when there’s a miss (block, mask, or route to safe path). if nobody wants to sample prod, at least do offline sampling from a mirror dataset with the same logging pipeline.
also, if the review form can’t express “data risk for sensitive strings in unstructured text,” then you don’t rely on checkboxes. you attach an addendum to the audit log saying what you approved vs what you didn’t, and you propose the missing control as a follow-up gate. i’ve helped teams turn that kind of audit-risk disconnect into a real lightweight gate so it doesn’t repeat
johnypita@reddit (OP)
thank you
this sounds exactly matched to our use case
ill try it out ,and also adopt this approach
Ok-Leopard-9917@reddit
You have a concern. Raise the concern with your lead, team members, the security team. What’s important is to raise the concern and build consensus on how it should be managed.
SansSariph@reddit
Processes exist so that when everyone is acting in good faith, they catch issues that would've been otherwise missed by mistake. They are often set up by people acting in good faith to plug gaps that have been missed before.
They do not work as designed when they become just a checklist in the way of shipping. People have to stop and think or they become theater. The trouble is they seem to always end up becoming "just checklists" when the people signing off are incentivized to ship first and foremost.
In your position I simply would not live with it until that uncertainty was cleared up. I would have (and have) refused to sign off given what seems to be a substantial privacy incident and required escalation to someone with authority to overrule my professional judgment. The optimistic view is I'm signing my name to ensure the right thing was done. The cynical view is I'm signing my name to absorb liability in the event of a problem. In both cases the lingering doubt keeps me from signing off.
johnypita@reddit (OP)
thank you
that is exactly what i needed to hear honestly
im going to adress this gap
cobalt-jam88@reddit
The existing review framework wasn't designed for LLM routing decisions, so it's not surprising three groups signed off on the wrong questions. In practice the gaps you're describing are consistent across this class of feature: no explicit check for the wrong-but-confident failure mode where the model routes with high internal confidence and neither downstream team has a reason to flag it, and no prompt injection assessment for user-generated input that feeds directly into the model context. The audit trail requirement is the other missing piece -- you need something that captures why a specific routing decision was made so you can debug it retroactively rather than just observe outcomes. The vendor checklist is checking for data-at-rest classification and network egress because that's what it was built for, not for automated decision accountability. Are you working from a shared review template across teams, or is each group running their own checklist in your env?
johnypita@reddit (OP)
thnks man
a shared one
CarelessPackage1982@reddit
care as much about it as the CEO cares about you
OldPurple4@reddit
On one hand, absolutely yes. But on the other, someone has to actually look out for the users. It is a pickle.
johnypita@reddit (OP)
i totally agree
this is exactly what i feel
bethanyw4ffle8865@reddit
what's the significance of the empty title?
ramenAtMidnight@reddit
Dude, never sign off on anything without your own criterias. It’s a bit too late now but I’ve seen engineers got thrown under the bus for this exact reason. Note this has nothing to do with AI or even security. Never sign off on things you are not certain about.
vilkazz@reddit
From security perspective, credentials appearing in tickets is already a security boundary breach. Human or LLM insesrted afterwards pose similar risk to the customers, and, i would argue, that humans are even higher risk than LLMs.
Therefore, as long as the required boxes were ticked elsewhere, I would likewise label the remaining risk as "low".
Instead, the credentials in tickets are double plus bad and I would further question whether reasonable effort is taken to clean them or to improve the ticket creation flow.
johnypita@reddit (OP)
can you explain why you think that the human breach is worse then the llm??
we we have access to much more sensitive sources of the company, the source code is a much more sensitive information of the company then a client credentials,
and the with the llm you can never know where does itc gets to
New-Locksmith-126@reddit
Except in rare circumstances, your source code is worth almost nothing to anyone but you.
Your customers data (yes including their credentials) are much more valuable. You should certainly never have access to unhashed passwords of your customers. If they send them to you they should be notified immediately and required to change them.
vilkazz@reddit
Humans can be subverted, access data through random channels (coffee shops, airport wifis), phished, spied on, so on so forth.
For LLM you can do due dilligence as with any software product to contractually ensure that your provider properly sandoxes your data and is not using it to train their next generation product. Note that this applies more to API calls and less to users pasting random stuff in their Claude, Copilot, whatever. Worst case, you can also run a local fully airgapped model.
in other words - human is a chaotic, unpredictable system, LLM is a software black box that can be more or less reasoned about
johnypita@reddit (OP)
i understand and agree, but stiill dont trust those companies to not use the data and not keep it , they had all massive lawsuits on privacy and data leaks, for my use case i egt your point
but generally i wouldnt want a company to share my private info to an llm even under an agreement
false_tautology@reddit
You wouldn't want your data in an LLM. You also wouldn't paste your authentication credentials into a plain textbox meant for a human to read. I may be callous to say this, but if a user is emailing their passwords, they're responsible for wherever it ends up.
johnypita@reddit (OP)
agreed
but we as the company devs should cover for them
and make sure that their data is safe even if they leak it
false_tautology@reddit
Then you need to up your redaction on input, not try to catch it downstream. You're seeing an LLM parsing the password as the problem, but that's a symptom of the actual problem.
If users are going to be pasting their passwords into tickets and it goes in the ticket, they're already compromised. I get trying to plug the leak, but if credentials are leaked it doesn't really matter how, and the LLM is the least likely place for that to happen.
If you want to solve it, you have to create a security audit process that better captures the passwords before they make it into your system. I'm going to mention, though, that the best way to catch those is going to be an LLM, ironically. Just make sure it isn't training off the data.
johnypita@reddit (OP)
thank s man
actually just looked into pii open source models that can provide this excat solution without involving and third party
do you know any pii solution that is best for this??
gefahr@reddit
OpenAI just open sourced one a few days ago that is claimed SotA for this. It's small enough to be self hostable (economically) too.
false_tautology@reddit
All of our redaction targets document data (PDF, PNG, etc.), so I'm not well versed in redacting text data itself.
We use proprietary tools like Azure detection models, which are very very good at this and we trust that Microsoft is holding to its contractual obligations that states that they will not train off of our data.
vilkazz@reddit
Fair enough
pl487@reddit
If you're at a place with that kind of security, surely you are already using AWS Bedrock or equivalent.
gfivksiausuwjtjtnv@reddit
The CISO specifically said that the risk was acceptable? It’s their call right?
PrintfReddit@reddit
You have an enterprise LLM API agreement, yes? At that point there is no real difference between an LLM or something like a S3. You should have legal protections in place that ensure the vendors can’t train on your data, about as much as any other vendor you might have. This is a non issue.
ButterflySammy@reddit
You wouldn't know about incidents, only reported incidents.
There's an important distinction.
jk_tx@reddit
Sounds to me like your real problem is idiot customers entering credentials where they shouldn't. You should do what you can to prevent such info from being entered into tickets in the first place, since any checks later in the processing pipelines are at best a band-aid.
I don't see how this as an AI issue, unless you think the humans that previously reviewed these tickets were better at catching and removing such credentials. But even if they were (doubtful), it sounds like human review would pose greater security risk than LLM review in this particular case. What's to stop the human reviewer from capturing/saving such credentials?
Esseratecades@reddit
Not completely sure I understand.
If I'm understanding you correctly then you signed off that someone checked whether there was the potential for data loss or not, and that said data loss potential is negligible, when really nobody checked that, including the people upstream of you.
Is that accurate?
johnypita@reddit (OP)
yes
Esseratecades@reddit
Oof, yeah that's not good.
Is there a process for reviewing for potential data loss? If there is a process then this is kinda on you for ignoring it. If there isn't, then you should at least develop a personal checklist or process that you can use. That will allow you to confidently and deterministically judge these kinds of things, such that if there is an issue you can say that there wasn't much you could've done.
johnypita@reddit (OP)
ive looked into pii models solutions, ill run some tests locally and check if i can mask the credentials that are gettting through
thanks
5olArchitect@reddit
Sorry is this an internal tool or not? How are customer creds getting into an internal tool?
need-not-worry@reddit
This looks like some shitty process already done, and you try to blame AI. What's different if a human read the sensitive data instead of LLM? Why would customers attaching sensitive data to the ticket? What's next, censor their messages if they post their credit card number?
WJMazepas@reddit
Just go on with your work, but be ready that, if this ever blow up and causes a big issue, to shift the blame to someone else and not you
johnypita@reddit (OP)
do you really think that they will put it on me ??
the security guys actually supposed to get the responsiblity for this, no?
WJMazepas@reddit
I dont know, but it is possible. Just be ready if something happens