Approved an AI feature for production knowing the security review didn't actually answer the question. It shipped, nothing happened, I still don't feel right about it.

Posted by johnypita@reddit | ExperiencedDevs | View on Reddit | 51 comments

The feature was an internal tool that routes support tickets to the right team using an LLM.

The security review went around for a week

Three groups signed off. The form had checkboxes for vendor approved, data classification, retention reviewed. All three got checked and I was the engineer of record so I got the final yes.

I sat on it for two days. The thing the form did not have a checkbox for was the thing I actually cared about

the ticket bodies sometimes contain customer credentials that customers paste in by accident. We strip the obvious ones but we dont catch all of them. Nobody can tell me what percentage we miss, because nobody has measured it, because measuring it would require sampling production tickets, which is its own approval cycle.

so I asked. The cisos office told me the existing controls were "appropriate to the risk class" when I asked what the risk class was I was told it was "internal tool with vendor under DPA" which was technically true and answered nothing

I approved it. The form got signed. The feature shipped. Six weeks later its still running and there has been no incident Im aware of

I gave a yes that I knew was answering a different question than the one I was actually being asked. The form I signed says the feature was reviewed for data risk. What I actually signed off on was that the review process had been completed. Those are different things and the form doesnt know the difference.

I described my own decision in the audit log as "approved per security review, low residual risk." The "per security review" part is doing all the work. I am pretty sure that is how every senior engineer in every company is currently approving every AI feature, and I know that is fine but still it feels off

Anyone else been the last signature on a chain that everyone is treating as "the review" without anyone having actually done the review? How do you live with it?