Lessons from deploying RAG bots for regulated industries
Posted by Neoprince86@reddit | LocalLLaMA | View on Reddit | 46 comments
Built a RAG-powered AI assistant for Australian workplace compliance use cases. Deployed it across construction sites, aged care facilities, and mining operations. Here's what I learned the hard way:
- Query expansion matters more than chunk size
Everyone obsesses over chunk size (400 words? 512 tokens?). The real win was generating 4 alternative phrasings of each query via Haiku, running all 4 against ChromaDB, then merging and deduplicating results. Retrieval quality jumped noticeably — especially for domain-specific jargon where users phrase things differently than document authors.
- Source boost for named documents
If a user's query contains words that match an indexed document title, force-include chunks from that doc regardless of semantic similarity. "What does our FIFO policy say about R&R flights?" should always pull from the FIFO policy — not just semantically similar chunks that happen to mention flights.
- Layer your prompts — don't let clients break Layer 1
Three-layer system: core security/safety rules (immutable), vertical personality (swappable per industry), client custom instructions (additive only). Clients cannot override Layer 1 via their custom instructions. Saved me from "ignore previous instructions" attacks and clients accidentally jailbreaking their own bots.
- Local embeddings are good enough
sentence-transformers all-MiniLM-L6-v2 running locally on ChromaDB. No external embedding API. For document Q&A in a specific domain, it performs close enough to ada-002 that the cost and latency savings are worth it. The LLM quality (Claude Haiku) is doing more work than the embeddings anyway.
- One droplet per client
Tried shared infrastructure first. The operational overhead of keeping ChromaDB collections isolated, managing API keys, and preventing cross-contamination was worse than just spinning a $6/mo VM per client. Each client owns their vector store. Their documents never touch shared infrastructure.
Happy to share code — RAG engine is on GitHub if anyone wants to pick it apart.
jannemansonh@reddit
the cross-tenant rag isolation point is real... spent way too long building custom namespacing. may want to give needle app a spin since it handles the RBAC on collection level
Neoprince86@reddit (OP)
I hear you - have live customers running on this infrastructure and it works :)
LoSboccacc@reddit
a couple missing things:
reranking is very effective, especially with a weak embedding model
small embedder and reranker gives the most ROI in accuracy per finetuning dollar spent
situational but it's often better to expand to the matched chunk to one of the containing paragraph, chapter or full document, before feeding the data to the llm. or at the very least if you get multiple chunk from the same document, present them in order
if you work at regulated industries enforce compliance boundaries at tool level. track which compliance a conversation has, and block tools that may leak information. (i.e. your llm can use open internet searches tool, but calling them after doing a search on private data results in a failure, and if you read internal policy you can still call tool to read customer cases, but not to write into them, etc. )
danmercede@reddit
I agree tool-level boundaries matter, but I think a lot of teams stop one layer too early.
Prompt rules are useful, retrieval filters are useful, tool permissions are useful — but none of those are really a hard constraint once the model is generating responses in a messy real workflow.
The bigger question for me is what sits between model response and application action. Is there anything there that can reject, quarantine, rewrite, or require approval when the output crosses a line?
Curious what people are actually doing in prod beyond prompt instructions and corpus filtering.
LoSboccacc@reddit
You kind of throw away the llm response, and tell the llm to call a tool that the application can convert into action, then you can apply whatever guardrail to that tool
SkyFeistyLlama8@reddit
Using a smaller LLM as a semantic filter also helps. Sometimes you could be dealing with multilingual documents or queries that a regular reranker would have a hard time with.
Neoprince86@reddit (OP)
Equivalent_Pen8241@reddit
Deploying into regulated industries definitely raises the bar for what 'production-ready' means. The security requirements often go way beyond basic data isolation, especially when you factor in sophisticated prompt injection and extraction risks that can lead to data leaks.
If you are looking for a robust way to handle these security layers, we have open-sourced SafeSemantics. It is a topological guardrail designed specifically for AI apps and agents. It plugs directly into your workflow to detect and neutralize penetration attempts using a deep knowledge base of attacker behaviors. It has been a game-changer for maintaining compliance and safety in high-stakes environments. Check it out: https://github.com/FastBuilderAI/safesemantics
caioribeiroclw@reddit
o ponto do droplet por cliente ressoa muito. aprendi da mesma forma -- shared infra parece economizar dinheiro mas o custo real e a complexidade de garantir que o contexto de um cliente nao vaze para o outro, especialmente quando vc tem historico de conversa sendo cacheado. a divisao que funcionou foi separar infra (compartilhada) de state (isolado por cliente). mas isso levanta uma questao que ainda nao resolvi bem: como vc versiona as atualizacoes do sistema prompt da camada 1 sem precisar fazer rollout manual em cada droplet?
Neoprince86@reddit (OP)
If I told you that I’d have to kill you 😂😂
caioribeiroclw@reddit
haha fair. though given you got this far with regulated industries, i suspect the secret involves more than just good prompt engineering 😄
caioribeiroclw@reddit
haha fair enough. honestly secrets in this space tend to get rediscovered independently within a few months anyway
eliko613@reddit
Great insights on the multi-tenant approach. The $6/mo VM per client strategy is smart for isolation, but I'm curious - how are you tracking LLM costs as you scale this across more clients?
With Claude Haiku for query expansion (4x calls per user query) plus the main LLM calls, the usage can add up quickly across multiple clients. I've found that without proper observability, it's easy to miss cost spikes or clients with unusually high usage patterns.
For regulated industries especially, having detailed logs of LLM interactions and costs per client becomes important for billing accuracy and compliance auditing. Are you handling that tracking manually or have you built something custom? We started testing zenllm.io for multi vendor visibility and optimization and it's been helpful so far.
The local embeddings choice makes total sense for cost control - that's often one of the first optimizations that pays off. Your architecture sounds solid for the current scale.
Neoprince86@reddit (OP)
Honest answer: we mostly sidestep the problem by design, but it's a real gap at scale.
How we handle it now: each client brings their own Anthropic API key — it lives on their droplet, they pay Anthropic directly. So there's no central billing pool to track. Cost attribution is zero overhead because it's not our bill. That's deliberate for the current scale and a good answer for SME clients who want cost ownership.
Where it breaks down: the calls we make — FrankQA test runs, provisioning-time indexing, the platform-level Haiku calls for RAG — those come from our key and aren't currently tracked per-client. If we start doing shared-key billing (e.g. enterprise tier where we absorb the API cost and charge a markup), we'd need per-client attribution fast.
What we'd build: a lightweight middleware layer that wraps the Anthropic client, tags every call with a client_id, and writes token counts + cost estimates to a SQLite or Postgres table. For compliance industries especially, we'd want request hash + response length logged alongside it for audit trail, not just cost. Then a simple dashboard that shows daily cost per client and flags anomalies (someone who usually uses 10k tokens/day suddenly using 200k is either a bug or a breach).
On zenllm.io: haven't used it — worth a look if you're aggregating across OpenAI + Anthropic + others. For single-vendor at our scale the custom approach is probably 2 hours of work and means we own the data.
What's your client mix on the regulated side — are they on shared key or BYO?
eliko613@reddit
Fair take for a static single-vendor setup — but the "2 hours of work" gets you logging, not insight. Context waste detection, prompt version drift, and anomaly baselines that distinguish growth from a runaway agent take real iteration. Most teams build the logging layer and never get to the rest.
The multi-vendor framing also undersells the problem — the hard part isn't tracking two providers, it's attributing a 3x cost spike across multiple agents and RAG pipelines on a single provider.
On data ownership — zenllm.io sits in the observability layer, not in your request path, so that concern doesn't apply.
For regulated industries specifically (the thread topic), having audit trails and reproducible cost attribution out of the box matters more when the person who built the custom SQLite schema has moved on.
SufficientTea8255@reddit
Your query expansion approach is solid. Honestly for 80% of compliance questions that's probably enough. I am curious if you've hit the multi-hop wall yet. Stuff like "does the FIFO policy override the fatigue management standard for night shift DIDO workers?" where the answer depends on how two documents reference each other, not just what they individually say, this is where things can get tricky.
i ran into this previously with regulatory docs that cross-reference each other constantly. Vector similarity kept pulling the right chunks from each doc separately but couldn't simply connect them. I ended up layering a relationship index on top of the vector store just for document-to-document links (references, supersedes, amends, applies_to, etc.). Not full GraphRAG, more like a lightweight graph that tells you which chunks need to be co-retrieved. Maintenance is the real cost though because the relationships shift every time a regulation updates
Neoprince86@reddit (OP)
That wall hit us hard enough that we built a whole layer just for it.
Vector hit rate was fine, but it was “close enough” in two different documents, and Claude couldn’t reason across them without being spoon-fed the connective tissue. So we added ATF (Action-Topology Format) on top of the normal chunks:
• During ingest we run each clause/page through Haiku to emit structured blocks with Data_Connections (references, overrides, applies_to, supersedes, etc.). • Those connections get persisted alongside the chunk IDs. At query time we retrieve the best chunk plus anything one or two hops away via those relationships and shove all of it into the prompt as a single stitched context window. • It’s still just Chroma underneath — no full GraphRAG server — but that lightweight graph is enough to co-retrieve “FIFO policy clause 12 references Fatigue Std 4.2, and the override condition says ‘night shift DIDO within WA mines.’”
Maintenance was the scary part, so we tied it to the document change pipeline. When SharePoint (or whatever source) shows a new version, we re-run ATF for only that doc, recompute the outbound/inbound connections for those chunks, and leave everything else alone. In practice it’s a nightly batch that costs a few dollars in Haiku calls and keeps the relationships fresh without a full graph rebuild.
Short version: query expansion + plain vectors got us to 80%. ATF + connection-following is how we got the weird 20% (FIFO vs fatigue, NES vs EBA carve-outs, WHS Act cross-cites) to stop hallucinating. Still not GraphRAG, but it’s enough structure for compliance questions to stay grounded.
o0genesis0o@reddit
How do you avoid client breaking layer 1 prompt?
Equivalent_Pen8241@reddit
These are fantastic lessons, especially the focus on multi-tenancy and isolation. In regulated sectors, the 'context leak' risk is huge. Reranking and chunk expansion help, but they don't solve the underlying problem that vector similarity is fundamentally probabilistic. We've seen teams in banking and legal start exploring vectorless ontological memory to get deterministic grounding that's 30x faster than traditional RAG pipelines. It's much easier to audit too. If you're interested in alternative memory architectures for high-stakes agents, check out FastMemory: https://fastbuilder.ai/fastmemory
adrq@reddit
How does this handle documents that are mostly tables? Thinking compliance matrices, policy tables, org charts etc. Paragraph-aware chunking makes sense but curious if it is able to retain row/column relationships and similar structured data?
Neoprince86@reddit (OP)
Tables are the hardest part. Paragraph-aware chunking breaks down completely when a row and its header are in separate chunks. Our current approach is imperfect: we detect table-like content during parsing (via pdfplumber's table extraction) and treat each table as a single chunk rather than splitting it. That preserves row/column relationships but means large compliance matrices blow out the chunk size and burn more context window. The honest answer is that structured data in PDFs is still a partially unsolved problem in RAG. We've had better luck asking clients to export policy tables as separate CSVs and indexing those separately with a structured query path alongside the vector retrieval. Not elegant but it works.
adrq@reddit
Yeah I've had good luck skipping the embedding for structured data entirely and having the LLM generate structured queries to grab data, interpret, and present a natural language response.
Equivalent_Pen8241@reddit
Regulated industries really push the limits of RAG, especially with accuracy and hallucination risks. One interesting alternative/complement for high-stakes environments is FastMemory (https://github.com/fastbuilderai/memory). It's vectorless and uses ontological structure, which significantly reduces hallucinations compared to standard vector retrieval. Plus, it's 30x faster in production. Worth looking into for those strict compliance use cases!
Neoprince86@reddit (OP)
Genuinly interesting - looking
mega-modz@reddit
How can I deal with same question multiple documents - like what is the average increase in revenue for year 2025 - the data may be in 5 to 6 pdf how can u sure it gets the user asked one ( u can't fetch 5-6 because it consumes so much context)
BeyondTheBlackBox@reddit
Idea 1: If this happens often - use an aux model to summarize the results if they amount over X tokens, X being the threshold.
Idea 2: Improve the data quality pre-ingestion to avoid this situation altogether.
Idea 3: If this happens not-so-often - accept the high token usage, make sure you cache the prefix and, honestly, the avg cost over a month won't be too bad in my experience
Idea 4: Use subagents + an agent rag via a filesystem, have the agents grep useful parts of the doc after retrieval (payload only includes the filename to query), then provide an answer with a main agent. Higher latency, very niche, but had some great results for complex queries
Idea 5: Use the vector store search as a tool instead of always appending the result to context. The llm can then do a multi-turn search and iterate until it finds all the relevant info or exhausts the search domain. A bit less trivial and requires thorough testing, but works really well for some tasks (in my experience, searching for learning materials in edtech space thorughout an internal database)
Idea 6 (supporting other ideas): Use a small fine-tuned llm as a router to choose which of the rag approaches to choose and how much effort to direct there.
Neoprince86@reddit (OP)
The multi-document aggregation case is one we've hit a lot. A few things that work well in practice:
For numeric aggregation specifically - the issue isn't just context size, it's that vector retrieval returns semantically relevant chunks but not necessarily the right data fields across all documents. What we've found works better is a two-stage approach: first pass extracts the target field from each document separately using a small structured extraction prompt, second pass aggregates the results. You never load all 5-6 PDFs at once -you run N cheap extractions and one cheap aggregation. Latency goes up slightly but token cost stays flat.
BeyondTheBlackBox's Idea 5 (iterative vector search) is underrated for this. We treat it as "retrieval with a scratchpad" - the LLM queries, records what it found, queries again with updated context, and stops when it has enough data or hits a budget. Works especially well when you don't know upfront which documents contain the answer.
The router idea (Idea 6) is where we're heading too - classify the query first, then choose the retrieval strategy. Simple factual = single chunk. Comparative = per-doc extraction + merge. Aggregation = structured extraction pipeline.
9gxa05s8fa8sh@reddit
I wonder if that applies to any or all other tasks. I assumed high-end AI tools were already doing pre-processing like this if it was effective, but apparently not.
it would be no big deal to take advantage of the AI bubble and have a pre-processor for your IDE send every prompt to 4 different free AI asking for different wordings that get merged together for the final prompt to send off...
CATLLM@reddit
Im trying to learn RAG, any tips for a beginner? Would love to see your code as well. Thank you!
BeyondTheBlackBox@reddit
I agree with the comment above and the top comment, however, wanna add some tips to it:
rorykoehler@reddit
RAG is simple as the tooling is built ... My pipeline is essentially
Creation: chunk document -> create embeddings -> store each chunks embeddings in postgres
Retrieval: Convert query to embedding using same embedding engine -> use similiarity search built into pgvector to find X number of similar chunks -> shove them all into the llm query and let the llm make sense of it.
Refinement possible as per OP etc
Neoprince86@reddit (OP)
Hey! Happy to share. The GitHub repo is here: https://github.com/LevelUpBM/frank-bot — the RAG engine is in shared/rag_engine.py, it's the real production code we run.
For a beginner, my honest advice: start with ChromaDB + sentence-transformers before touching any paid embedding API. Get chunking and retrieval working locally first - you'll understand why the parameters matter before you start paying for it. The repo has a standalone test you can run at the bottom of rag_engine.py.
Main things beginners usually get wrong: chunk size too large (loses precision), not enough overlap (breaks context at boundaries), and not filtering by relevance score (garbage chunks tank your answers). Start small, test retrieval quality manually before you hook up the LLM.
You can check out our public engine at www.askfrank.com.au as well :)
Ok-Drawing-2724@reddit
The three-layer prompt system is a solid idea. ClawSecure is useful for scanning RAG bots in regulated industries to catch potential issues early.
thrownawaymane@reddit
Can we ban this guy?
He just shills his open claw product in comments. There’s a respectful way to mention your project (and clearly disclose that it’s yours)
I haven’t seen you do it once.
No_Individual_8178@reddit
the query expansion approach is smart, i do something similar but trigger retrieval conditionally based on model output entropy instead of hitting the vector store every turn. saves a lot of redundant lookups when the model already has enough context from prior chunks. also +1 on MiniLM being fine for domain stuff, i run it with ChromaDB locally and honestly the embedding model choice matters way less than how you chunk and handle cross-document references. curious about your source boost implementation though, are you doing exact title matching or fuzzy? because i found that users misspell or abbreviate document names constantly and exact match misses like half the cases.
Neoprince86@reddit (OP)
The entropy-based conditional retrieval is clever, would be interested to hear more about how you're measuring that in practice. Are you looking at token probability distributions from the model or something simpler like confidence signals from the generation?
On source boost, we do a hybrid. Not exact title match, but not full fuzzy either. We strip annotation tags from source names, split into words over 3 chars, and require 2+ of those words to appear in the query (lowercased). So "Casual Employee Policy DRAFT.docx" becomes ["casual", "employee", "policy", "draft"] and we check how many hit the query string. Works reasonably well for abbreviations since "casual employee" usually survives even if the user drops "policy" or "draft".
You're right that it misses misspellings though. We haven't hit it as a major problem yet in our deployments because most users are querying by topic not document name, but for document-heavy workflows (legal, contracts, multi-version specs) I can see it being a real gap. Have you tried token-level overlap or edit distance on the word tokens? Wondering if that's worth the added complexity or if you just solved it with better onboarding (telling users the exact doc names)
No_Individual_8178@reddit
for the entropy thing it's pretty simple. i just check if the top token probabilities are spread out after the last generated chunk. high entropy = model is guessing = go fetch more context. if it's confident i skip the retrieval entirely. not perfect but cuts like 40% of unnecessary lookups which was good enough for me. your word overlap approach sounds solid, 2+ words is a good threshold. for misspellings i just went nuclear on normalization — lowercase everything, strip punctuation, sometimes stem. looked into edit distance but it gets expensive fast with hundreds of docs and the aggressive normalization already caught most of what i was missing so i just stopped there.
Luis15pt@reddit
What information does get sent to anthropic?
Neoprince86@reddit (OP)
Good question. Per request, Anthropic receives: the system prompt (bot personality + company instructions), the top-k document chunks retrieved for that query (not the full documents, just the relevant excerpts), the user's message, and the conversation history for that session.
What doesn't get sent: the full documents, the vector store, any user identity info, or the client's API key.
Worth noting: in our setup each client runs Frank on their own server using their own Anthropic API key. Their data goes to Anthropic under their own account. We never see their API calls at all.
Anthropic's API policy also doesn't use inputs/outputs for training by default, unlike the consumer Claude.ai (http://claude.ai/) product. So for enterprise clients asking about data handling, that's the key distinction to make.
Luis15pt@reddit
Why not self host a local model and keep everything offline ? Yes the customer would make and additional hardware investment but maybe that's something you could also support?
Neoprince86@reddit (OP)
Ability of local model to deal with complex reasoning - have one running basic tasks at the moment though
Nova_Elvaris@reddit
The query expansion point is underappreciated. In my experience the biggest retrieval failures come from vocabulary mismatch between how users ask questions and how documents are written, and generating alternative phrasings is one of the cheapest ways to close that gap. One thing worth adding to your three-layer prompt architecture: logging which layer triggered a refusal or override. When you're debugging why a bot gave a weird answer in production, being able to trace whether it was Layer 1 safety, Layer 2 vertical rules, or Layer 3 client instructions that shaped the response saves a lot of guesswork.
Neoprince86@reddit (OP)
Good point and actually we do have a 3-layer system in our HR/compliance bots already (security layer, vertical personality layer, client custom instructions layer). The tracing problem is real though. We know which layers fired but we don't log which layer shaped a specific response or triggered a refusal. That diagnostic piece is missing. Debugging a bad output right now means reading the full composed prompt and reasoning backwards, which is slow.
Max-_-Power@reddit
Underrated post IMO, thanks
Polite_Jello_377@reddit
Can you link to the GitHub repo?
Neoprince86@reddit (OP)
https://github.com/LevelUpBM/frank-bot