Local Arabic Legal Chatbot (RAG + LLM) – Need Advice

Posted by Maleficent-Town8242@reddit | LocalLLaMA | View on Reddit | 4 comments

Hi everyone,

I’m currently working on a project to build a 100% local AI chatbot for a government-related use case focused on data protection (DPO support).

The goal is to create a chatbot that can answer questions about legal texts, regulations, and personal data protection laws, mainly in Arabic. Because of the sensitive nature of the data, everything must run locally (no external APIs).

Current approach:

Using a RAG (Retrieval-Augmented Generation) architecture
Local LLM (considering LLaMA 3 or Mistral)
Embeddings with bge-m3
Vector database (FAISS or ChromaDB)
Backend with FastAPI

What I need help with:

What’s the best local LLM for Arabic legal content right now?
Any feedback on using bge-m3 for Arabic RAG?
Should I consider fine-tuning, or is RAG enough for this use case?
Any real-world examples of government / legal chatbots running fully local?
Tips to reduce hallucinations in legal answers?

Thanks in advance!

[-]

ComprehensiveBed5368@reddit

I think allam-7B-preview-V2 or Fanar-2.0-27B are good options for this task as well as: Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B Gemma4-31B Gemma4-26B-a4B

نموذج علّام السعودي ب ٧ مليار معلمة الاصدار الثاني v2 )متوفر فقط ك safetensor( والنموذج القطري الجديد fanar-2-27B كلاهما مناسبان للمهمة التي ذكرتها وكلاهما مدربان على بيانات عربية. بالنسبة للنماذج المفتوحة من شركات عالمية فأقترح Qwen3.5-9B Qwen3.5-27B Qwen3.5-35B Gemma4-31B Gemma4-26B-a4B

[-]

Maleficent-Town8242@reddit (OP)

Thanks a lot for these suggestions, this is really helpful 🙏

I’ve been considering models like LLaMA 3 and Mistral 7B, but your recommendation of Arabic-focused models like Allam-7B-preview-V2 and Fanar-2.0-27B makes a lot of sense for this use case.

The Qwen3.5 and Gemma models also look promising, especially for multilingual and reasoning capabilities.

Since the chatbot is focused on Arabic legal content, I’m trying to balance between:

Strong Arabic understanding
Good reasoning for legal queries
Ability to run locally (hardware constraints)

From your experience, do you think:
👉 Arabic-specialized models (like Allam / Fanar) perform better than multilingual ones (Qwen / Gemma) in a RAG setup for legal QA?

Also, have you tested any of these specifically with retrieval-based pipelines (RAG)?

Thanks again, really appreciate the insights!

[-]

InitialFox8963@reddit

is it only text or audio needs tobe fed as well ? into the chatbot

[-]

Maleficent-Town8242@reddit (OP)

For now, the chatbot is text-based only. We’re focusing on processing written legal documents (laws, regulations, guidelines) and answering user questions in text using a RAG pipeline.