How do I evaluate browser-based AI security without over-engineering it?
Posted by Any-Bet9069@reddit | sysadmin | View on Reddit | 15 comments
I’m an IT manager at a mid sized company, around 700 employees, mostly managed Windows laptops, Intune, Entra, normal web filtering and too many SaaS apps.
Our security team is getting more nervous about browser-based AI tools now. HR and marketing are using ChatGPT for docs, devs keep asking about Claude / Claude Code workflows, some people use Perplexity, some use Gemini, and I’m sure there are random AI writing extensions sitting in browsers that nobody approved.
I’m not trying to become the AI police. I also don’t want to be the guy who tells leadership “yeah we had a policy” after someone pasted customer data into a personal AI account.
So I’m trying to build a simple evaluation checklist before we buy another tool or just block everything and pretend the problem is solved.
The basic issue is this. If the laptop is managed, we can do some things with Intune, browser policy, web filtering, CASB/SSE, extension allowlists, etc. Not perfect, but at least there is a control path.
If the user is a contractor or on BYOD, it gets ugly fast.
Most AI usage happens in the browser, so normal network visibility does not always answer the question I actually care about. I don’t only care that someone went to chatgpt.com. I care if they pasted sensitive text, uploaded a file, used a personal account, used an extension that can read page content, or opened the same app from an unmanaged profile.
Things I’m checking so far:
Can we see browser-based AI usage clearly, or only domains/categories?
Can we separate approved AI tools from random shadow AI tools?
Can we control file uploads and copy/paste into AI tools without breaking normal work?
Does it work with Chrome and Edge, or only one browser?
Does it depend on a browser extension, and if yes can we actually enforce that through Intune?
What happens if someone uses a personal Chrome profile, guest profile, or another browser?
Does it help with AI extensions and permission changes, or only normal web traffic?
Does it support SAML / Okta / Entra properly, or are we creating another login mess?
Can we apply different policies for employees vs contractors?
Can we secure access for unmanaged devices without installing agents on personal laptops?
How noisy is the reporting? I do not want another dashboard full of alerts nobody reads.
What happens if we cancel, do we get logs/export, and how long do they keep the data?
Right now I’m seeing a few categories and none of them feel perfect.
CASB/SSE helps with broad visibility and policy, but sometimes feels too far away from the browser action.
Browser extension tools seem useful if you can enforce the extension properly, but that depends on how clean your managed fleet is.
Enterprise browsers seem strong if you can force users into the browser, but I can already hear the complaints from devs and contractors.
Agentless SSE / secure web access tools look interesting for contractor and unmanaged device access, because they focus more on securing the session/access path instead of owning the endpoint, but then I assume you give up some local machine telemetry.
I’m not looking for vendor pitches. I want the checklist from people who already had to deal with this.
What did you check before approving browser-based AI tools, and what did you miss that became painful later?
Josh_Fabsoft@reddit
Your approach sounds solid. The browser policy + CASB/SSE combo that ManyIndependence5604 mentioned is probably your best bet for the scale you're dealing with.
A few practical additions to your evaluation:
Shadow IT discovery first - Before locking anything down, run a quick audit of what AI tools are actually being used. Most CASB solutions can give you this visibility without much setup. You might be surprised what's already in use.
Pilot with power users - Start with your dev team and marketing folks who are already pushing boundaries. They'll surface edge cases and workflow breaks before you roll out company-wide.
Data classification matters more than tool blocking - Focus policies on what data can go where rather than blanket AI tool restrictions. Public marketing copy going to ChatGPT? Probably fine. Customer PII or code? Different story.
Consider approved alternatives - If people need AI for legitimate work, giving them a sanctioned option (even if it's just ChatGPT Enterprise with proper controls) reduces shadow IT pressure.
The enterprise browser route works well if you can stomach the user friction. Edge for Business with proper policies can contain a lot of risk while still letting people work. Just make sure your CASB can actually see and control what you think it can see.
What's your current CASB solution? Some handle AI tool visibility better than others.
That_Lemon9463@reddit
two gaps i'd add to the checklist after going through this last year.
(1) BrowserSignin policy. on managed chrome, set `BrowserSignin: 2` (force corp google sign-in) plus `RestrictSigninToPattern` to your domain. without this, users can swipe to a personal profile in the same chrome window and your extension allowlist + casb policy go away. most evaluation demos miss this because vendors test on a single signed-in profile.
(2) ask every vendor: "when an incident hits and i need to know what was pasted, do you log content or only metadata?" extension-based ai tools (layerx, push, nudge) tend to log metadata only by default. dlp-grade inspection (symantec, forcepoint) actually retains the prompt body, but with all the privacy/works-council headaches that come with it. you have to pick which incident class you can answer.
contractor/byod side: stop trying to own the endpoint. clientless reverse-proxy (cloudflare zero trust browser isolation, island in remote mode, or citrix secure browser for the picky cases) lands the user in a server-side chrome you fully instrument. you give up local file-upload telemetry, but you stop having "is this even chrome 130 with manifest v3 enforced" as an open question.
last thing on noise: filter your eval to "tools that distinguish personal vs SSO login on the same domain". most products bucket all chatgpt.com hits together. push security and a few others split it out. that single signal cuts alert volume meaningfully because legitimate enterprise-account use stops triggering review.
Myriade-de-Couilles@reddit
You weirdly already have at least some of the answers but don’t want to hear them.
> Enterprise browsers seem strong if you can force users into the browser, but I can already hear the complaints from devs and contractors.
You can let the door open or you can make some people unhappy that’s it’s closed, pick one.
The actual technical implementation will depend on what you want to open already but here is what we do:
- Corporste devices only
- Contractors and BYOD access via AVD
- Edge enforced and other browsers blocked (Applocker)
- Copilot allowed as it stays within our data governance perimeter but other AI blocked (Edge web filtering is very useful for that)
- Claude Desktop allowed for devs and a few other special cases but configured to use Azure Foundry so data stays again in our control
Competitive_Smoke948@reddit
Menlo security have basically a proxy that any browser will work with & opens in a remote browser in a container in their DC.
If developers & contractors have an issue I'd feel free to tell them to fuck of and get another job. virtually EVERY day hack & leak has been 3rd party developers or contractors using their own laptops.
WolfetoneRebel@reddit
Menlo is great but prepare your brain for never ending split-brains.
Competitive_Smoke948@reddit
have you seen push security?
caliber88@reddit
Meaning what?
plump-lamp@reddit
Webfilter. Block all generative AI except copilot because I assume you have at least an e3 license? Auto sign in edge and it'll be data protected.
kanaarei@reddit
You’re not overthinking it. You’re just far enough in to see where it breaks. Most of these tools answer the wrong question. They’ll tell you someone went to ChatGPT or Claude. They won’t tell you if it was a company account or someone dumping data into a personal one. That gap is the whole problem.
From there it unravels pretty quick. Controls assume one browser, one profile. That’s not how people work. They hit friction, open another browser, and keep going. Not malicious, just normal user behavior... but now your controls don’t mean much.
Upload blocking is easy. Clipboard either annoys people or doesn’t actually stop anything. Extensions are usually a total blind spot if teams aren't managing them with enterprise controls. Tons of AI helpers out there that can read page content and nobody’s really tracking them, feels like every tool has an AI agent baked into it now that can help you with whatever task you have. One of the most egregious examples of this recently we saw was an HR platform that basically wrote managers reviews, goals, etc for them... totally uncontrolled and unauditable.
Don't even get me started on the problem with contractors and BYOD type scenarios... nightmare.
What actually got us wasn’t missing features, it was realizing how much was happening outside the stuff we approved. We had licenses, policies, all of it. Still had a ton of shadow AI right next to it.
I know you're not looking for vendor pitches... but I'm gonna pitch one anyway. This whole issue is what pushed us to build KAiZAI.io. We couldn’t find anything that tied identity, browser behavior, and policy together in a way that held up in real use... so we built it. Check it out if you're interested, it definitely could help with some of the issues you're seeing.
When you’re looking at vendors, I’d just keep it simple. Can you tell who the user actually is inside the tool, not just the site? Can you separate approved use from everything else? Is it easy to get around? And if something goes wrong, do you actually have useful logs?
If those answers are vague, it’ll look good in a demo and hurt later.
Helpjuice@reddit
Kill the BYOD and contractor laptops your cooked by allowing this and cannot properly secure or manage anything this way. Only allowed company owned machines on the network.
In terms of insight you should only allow Google Chrome, Firefox, and Edge and manage those browsers by only allowing whitelisted extentions.
You should be using modern EDR and DLP which also have their own browser extensions that allow you access to all browser actions and capabilities e.g.s, keystrokes, websites traffic unencrypted, screenshots, etc.). This is why only managed corporate machines should be allowed because you can and should have full access to what is being done on them.
You should have sysmon and audit logging enabled so you know all commands run on user machines. The bulk of users should be non-admins and admins should not be running any of the AI controlled applications on user machines. All apps should be deployable through software center or alternative enterprise managed software approval and distribution means.
Also note you being an IT manager there should be more work being done by security and not IT for this tasking you have created for evaluation of the security of an AI browser extension, or any extension or software. This should involve actual security engineers (AppSec Engineers, Reverse Engineers, Threat Intelligence Engineers, etc.) and not be something IT is expected to drive. This way you stay doing management and not getting into the weeds so you are more effective at getting the big picture and the resources you need to create and manage a team properly get the IT components done properly for this to include testing infrastructure for the security team for continuous testing and evaluation workflows and pipelines.
How do I know this works, because it is what we do. We have a security org that goes through all of the code of the extensions for each browser for each update along with integrated automated evaluation solutions. This is a necessary evil due to the malicious uptick in extension highjacking and supply chain exploits. Trying to do this 100% manually will cause a slowdown in the business so you'll want to automate as much as possible and the engineers can then dive into findings. This has allowed catching of popular extensions when they had supply chain issues to never get through to impact even dev. This will also give you an IT a full understanding of what is included in the extensions, what they call out too, when they do what and see what has changed through every update. If you don't have that level of technical people on your team you can use AI to summarize the changes into a narrative along with give you an update on what the security team has done.
Either way have security do the vetting, you power the tech to help make it happen even if that is just building out test and evaluation infrastructure.
Short-Legs-Long-Neck@reddit
But arent you guys are talking tools. Should you have a governance framework that says, these are the rules for these types of data? Essentially dictating the circumstance data can traverse various boundaries?
Sounds a little bit like the OP is in the age old 'how can i make this safer' mindset. In my opinion, its endless, and never really safe.
I am in the same boat as the OP, but pushing for the framework now.
slackmaster2k@reddit
You seem to be headed in a good direction long term. However, be mindful of putting frameworks and models and paperwork ahead of actual progress. If you can observe a risk and can control it now, just do it.
The parent sort of overlooked this OWASP resource: https://genai.owasp.org/
iamoldbutididit@reddit
I love your question but I'm concerned that you're approaching it from the wrong end. Many of the items on your list are controls. Controls are supposed to address risk. Risk is supposed to be quantified, evaluated and accepted or mitigated by management according to existing policy frameworks. If this isn't done you'll end up with some VIP telling IT that the controls they have in place shouldn't apply to him, his special employees, or his entire group.
Maybe you've already done this, but for those who are just jumping to controls, here is how its supposed to happen:
The risks of AI (there are many) are documented and quantified in a risk register. A risk board reviews the register and decides what action to take for each risk. If mitigation is chosen, controls are selected which lessen the risk. The controls, once implemented, are then monitored for effectiveness. Identify, analyze, decide, treat, and monitor, are what are know as the risk lifecycle.
I know many of us are in the same boat, and I am getting the sense that the only real answer is to shift to an allow-only internet access. Why does an intern in accounting need access to google? If their job entails taking invoices and entering the numbers into a spreadsheet they probably don't. What about someone in shipping who only ever uses purolator.com? Why can they go to any other website? The reality is that it will suck to have to manage, but shifting to a whitelist approach to the internet might be one of the best controls to prevent all sorts of bad things like malware (ransomware), data loss, and C2.
MeetJoan@reddit
The thing nobody talks about until they're cleaning it up: AI prompt history persistence on personal accounts.
The thing nobody talks about until they're cleaning it up: AI prompt history persistence on personal accounts. Even if you block uploads, paste, and shadow tools perfectly, if someone used their personal ChatGPT account once and pasted in customer data, it's still sitting in OpenAI's training-eligible logs unless they explicitly opted out. Worth adding "do we have a way to detect personal vs SSO logins on AI tools" to your checklist.
Also missing from your list: what happens with copy-paste out of AI tools. Most evaluations focus on data going in, but devs pasting AI-generated code with embedded API keys or credentials back into your codebase is its own category of pain.
Otherwise the checklist is solid. The thing that bit us hardest: noisy reporting. We picked a tool that surfaced every AI interaction and the alert volume meant nobody looked at any of them within two months. Whatever you pick, demo it on a representative team for two weeks before signing - the signal-to-noise ratio matters more than the feature list
ManyIndependence5604@reddit
I think your checklist is mostly right, but I’d add one filter before looking at any vendor:
who owns the browser session?
If it is a managed employee laptop, then Intune + browser policy + CASB/SSE + maybe a LayerX type extension can work, assuming you can actually enforce the extension and stop users from just moving to another profile/browser.
If you want very deep control, Island/Talon type enterprise browsers are probably stronger, but then you are buying an adoption project, not only a security tool.
Where I’d be careful is contractors and BYOD. That is where a lot of these models start to feel fake because you do not own the endpoint. For that group I’d look at Red Access https://redaccess.io/ or similar agentless secure web access tools, because the control point is the access/session path, not “please install our thing on your personal laptop.”
Not the same telemetry as owning the browser, but probably a more realistic deployment model for unmanaged users.