If Accuracy > Efficiency, How Would You Spec A Local RAG Machine?
Posted by elgringorojo@reddit | LocalLLaMA | View on Reddit | 11 comments
Hey all,
I’ve already built a proof of concept on my personal machine (4090 + 64GB RAM) for a fully offline setup handling medical-records Q&A and drafting from my own documents, and it works well enough to show the idea is viable.
Now I’m trying to spec a real dedicated office machine. One key requirement is handling a few PDFs totaling 1000 - 1500ish pages. (maybe 1m tokens I think?) (Sometimes more but rarely) I understand this is fundamentally a RAG problem rather than fitting everything into context, but precision really matters here (medical records), so I’m even considering more brute force approaches if hardware can support it.
For those running more serious local setups, is sticking with a single 4090-class GPU still the best value, or does this kind of use case justify moving to higher VRAM or multi-GPU? And if you’ve prioritized accuracy over efficiency, where did you see the biggest gains or bottlenecks?
Ive been playing around in my head with repurposing an old 3080 I have to do the chunking and then get an RTX 6000 ADA 48gb but is that over kill? Would an rtx 6000 blackwell be able to hold that much in context for brute forcing?
Would really appreciate any real world experience here
croholdr@reddit
no real world experience with 4090 class or blackwell; but 1 m tokens in context as accurate as it will allow; will need more than one blackwell plus a 4090 plus 64 gb ram. you could use up a h100 with that workload.
also multigpu past two gets real expensive real fast on a ddr5 platform
elgringorojo@reddit (OP)
Yeah I found a MB that has a pcie 5 slot but is also lga1700 and ddr4 so that’s my current plan given ram is like Gpu prices
croholdr@reddit
im probably going to end up with three computers running ubuntu and a fourth computer as a nas. and a mega inference tent in the garage running 14 cards at pcie v3 @1x.
so im just trying to get the networking / smart home in order cause some computers will be on wifi (hopefully 6E/7) ideally im trying to remote shutdown/wake on lan in an automated fashion based on AI workload.
i tried putting them in one room; it sucks. summer is coming and I dont have AC and they get hot.
im looking at 870E boards to run two 5070ti. There's an asus rog board that has 2 16x v5 pcie slots; of course you could only run that at 8x/8x. Its only 599$. But that sure beats pcie 4.0 at any speed (other than 16x) so I feel thats a bit easier than going the server mb route.
elgringorojo@reddit (OP)
Nice! I found this guy for 200 bucks https://www.asus.com/motherboards-components/motherboards/tuf-gaming/tuf-gaming-z790-plus-wifi-d4/
croholdr@reddit
dude it only has one 5.0 slot, the second is 4.0@4x
nopanolator@reddit
Accuracy > Efficiency don't make much sense to me. I don't get the point in the context.
>> One key requirement is handling a few PDFs totaling 1000 - 1500ish pages.
You should start by this before, using PDF format is suboptimal. Better to use an unified markdown format sliced in categories or a SQL server for this.
>> justify moving to higher VRAM or multi-GPU?
It will depend more on the model you're using and how you're using it (context margin required)
elgringorojo@reddit (OP)
So I meant that with a lower weight model or quantization the precision of answers declines. And then the limitation of RAG where if the RAG model doesn't serve the right chunks, the main model is blind to some info. With med records this could turn a 'right' hand into a 'left' hand somewhere along the way or like amlodipine into amitriptyline or something like that. That kind of accuracy is more important to me than cost or speed. Id rather leave it to digest records over night and then be 100% correct which is why I think brute force would be good if I didnt have to spend 30k on GPUs
nopanolator@reddit
Got it, thx for the details.
I rarely go below Q8 for production, compression is rising fast the error rate. And you have zero margin due to the specificity (i don't have experience on such critical case, i extrapolate from a mechanic case).
It don't change the starting point of the equation : to know your real need in VRAM you have to start with the size of the model (i'll try a fews to find the most relevant) and how you're planning to prompt it.
If it's just a crawl of the document to find a reference, you don't need a big model. But if it's to "talk" with the model about the said document, you will need a given margin with the context window.
What you're calling "brute force" is problematic only by the methodology applied and the format used. On a similar case : car's RTAs. Generally you want to have a set per brand > per model > per engine with an quick index as a compass for the model. Then eventually a real time list of parts and their providers on a SQL base.
You should start by there in my opinion : curating your datas to be compliant with the constraint. Taking one those PDF and to convert it in markdowns files (it's already a big reduction of errors and VRAM consumption) then slicing the said PDF by relevant chapters or items. From there, you create another markdown that is an index loaded to the model. So your model have only the indexes of the documents loaded and it know directly what file to read (and don't read) considering your prompt.
And to train your model (qLoRA is fast and cheap) on your specific use, once you find the more reliable for you. I've never tested but there is a fews "Health" Gemma models available on huggin face that can maybe help for relevancy and accuracy on complex prompts.
elgringorojo@reddit (OP)
Yeah that makes sense and when I rework the stack I'm going to play around with a lot of this. I guess at this point I'm trying to figure out if it's worth spending more for the rtx 6000 blackwell
nopanolator@reddit
Do it. You're far enough wealthy to make me a sweet price for you unusable 4090.
nopanolator@reddit
God it happens again, i'm too curious and i fell in the trap. Forget all i said, you absolutely need a couple of H100 to read a pdf lol what a fcking waste of time. lesson learnt.