I might use it as the LLM, but my RAG implementation doesn't use an embedding model, and my usual LLM for final inference is Gemma3-12B or Gemma3-27B.

My RAG implementation uses a HyDE step before traditional LTS via Lucy Search, which indexes documents as text, not embeddings.

The HyDE step helps close the gap between traditional LTS and vector search by introducing search terms which are semantically related to the user's prompt.

Lucy Search then retrieves entire documents, rather than vectorized chunks, the top N scored documents' sentences are weighted according to prompt word occurance, and an nltk/punkt summarizer prunes the retrieved content until the N documents' summaries fit within the specified context budget. This gives me a context much more densely packed with relevant information, and less relevant information lost across chunk boundaries.

That summarization step with that technology precludes the pre-vectorization of the documents, but with a lot of work it should be possible to make a summarizer for vectorized content. So far I haven't found it worthwhile to prioritize that work.

The summarized retrieved content is then vectorized at inference time, and final inference begins.

I'm pretty happy with the quality of final inference, and Lucy Search scales a lot better than any vector databases I've tried, but it's not without disadvantages:

• The HyDE step introduces latency, though I'm hopeful Gemma3-270M will reduce that a lot (been meaning to try it),

• My sentence-weighting algorithm lacks stemming logic, so sometimes it misses the mark; I've been meaning to remedy that,

• nltk/punkt is pretty fast, but also introduces latency in the summarization step,

• Vectorizing the content at inference time adds yet more latency.

So overall it's pretty slow, even though Lucy Search itself is quite fast. Everything else gets in the way.

My usual go-to for the HyDE step is one of Tulu3-8B, Phi-4, or Gemma3-12B, depending on the data domain, but I'm looking forward to trying Gemma3-270M for much faster HyDE.

My usual go-to for the final inference step is either Gemma3-12B (for "fast RAG") or Gemma3-27B (for "quality RAG"). Its RAG skills are quite good, and its 128K context accommodates large summarized retrievals, though I find its competence drops off after about 90K. My default configuration only fills it to 82K with retrieved content and the user's prompt, leaving 8K for the inferred reply.

I will be publishing my implementation as open source eventually, but I have a fairly long to-do list to work through before then.