Local AI for a small/median accounting firm - € Buget of 10k-25k
Posted by AFruitShopOwner@reddit | LocalLLaMA | View on Reddit | 139 comments
Our medium-sized accounting firm (around 100 people) in the Netherlands is looking to set up a local AI system, I'm hoping to tap into your collective wisdom for some recommendations. The budget is roughly €10k-€25k. This is purely for the hardware. I'll be able to build the system myself. I'll also handle the software side. I don't have a lot of experience actually running local models but I do spent a lot of my free time watching videos about it.
We're going local for privacy. Keeping sensitive client data in-house is paramount. My boss does not want anything going to the cloud.
Some more info about use cases what I had in mind:
- RAG system for professional questions about Dutch accounting standards and laws.
- Analyzing and summarizing various files like contracts, invoices, emails, excel sheets, word files and pdfs.
- Developing AI agents for more advanced task automation.
- Coding assistance for our data analyst (mainly in Python).
I'm looking for broad advice on:
Hardware
- Go with a CPU based or GPU based set up?
- If I go with GPU's should I go with a couple of consumer GPU's like 3090/4090's or maybe a single Pro 6000? Why pick one over the other (cost obviously)
Software
- Operating System: Is Linux still the go-to for optimal AI performance and compatibility with frameworks?
- Local AI Model (LLMs): What LLMs are generally recommended for a mix of RAG, summarization, agentic workflows, and coding? Or should I consider running multiple models? I've read some positive reviews about qwen3 235b. Can I even run a model like that with reasonable tps within this budget? Probably not the full 235b variant?
- Inference Software: What are the best tools for running open-source LLMs locally, from user-friendly options for beginners to high-performance frameworks for scaling?
- Supporting Software: What recommendations do you have for open-source tools or frameworks for building RAG systems (vector databases, RAG frameworks) and AI agents?
Any general insights, experiences, or project architectural advice would be greatly appreciated!
Thanks in advance for your input!
Disastrous_Grab_4687@reddit
How about you just queue the inferences and serve one (or more depending on your hardware) inferences at a time? You could get by with smaller hardware.
About RAG: I haven't had a good experience with ragging documents with Mistral small instruct 24b. The LLM still hallucinates sometimes. And it is not easy to rag resources in an efficient way so the LLM knows what to answer..
TechnicalGeologist99@reddit
Depends...
Azure gives access to open AI models that I believe they guarantee is hosted on your own azure infrastructure. So this is one off the shelf solution that will save you reinventing the wheel.
Personally I see a lot of potential for SMEs to use the upcoming ming DGX Station to provide inference for a couple 100 users. Could quite nicely host a few NIMs (Nividia Inference Modules) on that system due to its high GPU memory of 288gb. That's enough to host a primary LLM, an embedding model, reranker, OCR all the RAG stuff. GB300 has 8TB/s of GPU memory bandwidth, probably get you 500 tokens per second tokens/per sec on middle sized models in INT4 (more if using MOE).
Deploy with kubernetes and you could have that system be a single node. could scale, up if workload increases, to cloud or other on prem devices.
But personally I'd say start small on infrastructure you own, could even get the DGX spark (confirmed $4000) as a development environment.
I've heard DGX Station system will cost between 20k (I heard this from a Dell sales person) up to 40k. Personally I think 40K matches the pricing of their GB200 system well.
Anyways you could feasibly set up an inference server on that system that serves a limited pool of users.
Noxusequal@reddit
If you have 20k to spend i would honestly look at used Workstations from shops like bargain hardware (uk based but honestly great deals) get 2 6000 pro cards. In a server with dual full 16x slots and idk 128gb of ram foe good measure. The level of Jack you need to reach that level of vram with consumer hardware is maybe not what you want in production thata why i am saying this. Then you can run mistral large as q8 or the big qwen moe as a q4 which both are very capable local systems for your tasks.
Maleficent_Age1577@reddit
depends on price, new 1 x rtx 6000 pro is about 9k.
Noxusequal@reddit
Yeah so 1000 left for a used Workstations should work. For llm stuff everything but the GPU is kindoff unimportant. Maybe having good amounts of storage.
Maleficent_Age1577@reddit
Your math isnt correct. 25k - 9k x 2 is 6k.
Noxusequal@reddit
Well it said 10-25k so I thought going for something in the middle but sure yes if you have 7k left over no reason not to go for the fastest underlying platform but the platform doesn't really matter for inference performance
Maleficent_Age1577@reddit
Of course it does. 2 highend cpus with highend ddr process faster than 1 lowend processor with lowend ddr?
Platform does matter.
Noxusequal@reddit
Not if the model is already loaded in vram and doesnt spill all the processing is done on the cards. Only thing that might is pcie speed if the model is split between cards. The rest only matters for loading or if the modell spills out of vram then it matters a great bit.
Maleficent_Age1577@reddit
"You should buy 2 x processor with fast ddr memory to use coming up bigger models too."
What part you didnt understand?
vibjelo@reddit
If OP has any technical chops at all, and know/can learn how to build their own PC with separate hardware, then that's highly preferable in terms of cost compared to buying those pre-packaged workstations that usually carry a pretty large markup.
Noxusequal@reddit
I honestly belife that that specifix side is pretty price competetive if you want workstation Hardware. So at least 2 full pcie x16 slots qith actually 32 lanes each.
But yes if you wanna go and hunt dowm individual parta for best prices you ca n be a bit cheaper but to give you an idea. A 128gb ram 20 core xeon workstation with a big 1200w power supply sita arround 400-500£ which is pretty solid in my opinion.
taoyx@reddit
Rather than gfx cards you could also use a mac. I suggest you get acquainted with LM Studio first then rent different setups for a while before purchasing.
Ceret@reddit
Sounds like you need to pay a consultant to advise on all of this properly. Maybe someone here can offer their services
AFruitShopOwner@reddit (OP)
Actually my boss is encouraging me to experiment, she wants this kind of knowledge in the company. Hiring outside consultants kind of defeats the purpose. I'm just here looking for some basic advice to help get me started. The most important thing to get right at this point is the hardware. I don't want to build a dead end system
Dependent_Status3831@reddit
It doesn’t have to. Not offering anything here but as a Dutch AI consultant with over twelve years of experience I can already tell you that in this entire thread , no one brought up the most important and common issues for what you want. It mostly has to do with our language. Good luck 👍🍀
After-Cell@reddit
VPC Virtual PC. Not the same as the cloud various levels of private.
Fast-Satisfaction482@reddit
I think that's a very good strategy for your company as long as your management is prepared to suffer a few setbacks before succeeding.
AFruitShopOwner@reddit (OP)
They are, I've been very clear about this. I personally think a prive cloud provider is the way to go but managmeent really doesnt want it. Fine by me, I'm now getting a stupid high budget to tinker with some cool hardware and I'll learn a thing or two in the proces
Fast-Satisfaction482@reddit
My personal advice would be to focus not only on total VRAM size, but also sufficient throughput. If you have 100 persons accessing open webui in parallel at random times, you need a serious amount of total tps for it to be usable.
If it's more like a batch processing system, where you run workflows overnight, maybe it would be smart to get the biggest, most intelligent model running that you can in order to obtain real insights on selected workloads. But if you basically want local chatgpt, you need loads of processing power. For this use case, I prefer mistral small. On my dual 4090 workstation, I get plenty of tps and context up to 128k. But I'm not sharing it with other users, so I don't know if that is enough tps for a shared openwebui instance serving 100 people (probably not).
AFruitShopOwner@reddit (OP)
Thanks, that is helpfull
Maleficent_Age1577@reddit
It depends what matters you as a company? Is it speed and energy efficiency or cheaper and more energy spending.
with 25k you can get 2 x rtx 6000 pro and pc around those. that would give 192gb of vram which is pretty much enough. that would be same as 8 x 3090s.
AFruitShopOwner@reddit (OP)
What matters most is accurate answers. Speed is a plus too but I could also set up a system that works via email instead of a chat, where the user doenst instantly get their answers. bacht/chunk processing.
I think I'll start out with one pro 6000, maybe I'll get another one if we are running out of capacity or want to run larger models.
bsnexecutable@reddit
One thing that I do know about RAG based applications is that you can never be 100% sure about the things that LLM spits out (applies to any model out there). One thing that you could do is make use of an application that lists sources along of with its answers and MAKE sure your employees that use it go through the sources once to confirm any facts or figures the model is summarizing. This is why I think its very risky when it comes to law and other critical areas.
refer to this huge database of LLM fuckups: https://www.damiencharlotin.com/hallucinations/
vibjelo@reddit
That's the thing with probabilities, kind of comes with the territory when dealing with ML and LLMs in particular :)
synthphreak@reddit
Yeah. You could argue that basically every model “hallucinates”, even non-language models and models which are discriminative not generative. Never thought about it this way, but it seems reasonable to me.
For example, show an image of a cat to an MNIST classifier and it will confidently tell you the image contains a handwritten digit… The mechanism behind this “hallucination” is quite different from how/why a generative LLM hallucinates. But the end result is exactly the same.
Environmental-Metal9@reddit
I mean, humans hallucinate all the time. But the reasons why that happens in humans are myriad… lack of sleep, misremembering something vividly, Deja vu, etc, are all examples of what we navigate mostly ok on a daily basis but never think about it as such. We could use the strict definition of hallucinations but then that wouldn’t apply to LLMs anymore either.
I guess it’s just jarring when we encounter an entity that so confidently states something as true and yet be so wrong (or worse, subtly but undetectably wrong) without any markers that we know to look for
Substantial-Hippo165@reddit
Actually for AI the word hallucinations is a misnomer. Let me explain:- The way LLMs are designed is to be a next token prediction engine where they discriminate between the highest probability top estimate and the rest are discarded. This means the farther away the predicted token is from the first input token, the LLM will loose context and start outputting (rather very confidently) things that are wrong/incorrect/non existent. It comes with the premise actually and how they work, so very little can be done to prevent this from happening 100% time. The best bet (like a few good suggestions) is to have it output the reference and have the users validate the results. It can be a very good assistant - just like any good assistant, the manager needs to validate. Hope this helped.
ieatrox@reddit
also true of an intern tasked with the same requests.
it's a scale of magnitude or more cheaper to build an ai for the initial requests and QC responses than to task people with busy work and expect they QC their own work.
You're not wrong, but there's no perfect alternative.
AFruitShopOwner@reddit (OP)
Thanks, I'm well aware of this risk. In my opinion all of my colleagues fall into one of two catogaries. The ones who blindly trust everthing the AI spits out and the ones who don't trust AI at all. This will probably be the hardest part of this entire project
Maleficent_Age1577@reddit
You should never trust 100%. Even the most expensive models like chatgpt tell sora isnt available for me in Europe. When i correct those they be like yeah bro, you are right and just keep going like nothing happened.
And if i start a new conversation the same error occures. So always check the answers if it matters monetarily or otherwise.
vibjelo@reddit
Just noticed this now, but hilarious that a webpage about LLM "hallucinations" seemingly get the details wrong themselves, the first listed case of " Rochon Eidsvig & Rochon Hafer v. JGB Collateral" did not have the outcome as listed in the table on that website :P
Sorry-Hyena6802@reddit
if your interest is batch processing, please look into VLLM as serving framework. it’s pretty much made for batching queries as efficiently as possible for throughput. just generally have enough kv cache to hold your requests and the estimated output length and you should be in a good spot there. so no using 80gb of vram on a model and hoping the last 16 gigs serves more than 50 users on agentic or multi QA conversations well.
coding_workflow@reddit
And AI validating AI is like having the criminal brother/family in the jury....
krileon@reddit
Then don't use an LLM, lol.
I would not use this for anything other than summarizing documents, maybe helping with emails, maybe dumbing down communication with clients. I would never use this in accounting. An LLM can't even do calculations correctly. It often gets basic addition and subtraction wrong.
After-Cell@reddit
There need to be a policy. For myself, I aim to use it for brainstorming ideas that I then follow up on reading citations.
Likewise, putting more in than out is a great rule of thumb.
There should be more. An ai policy for the careful
AnomalyNexus@reddit
You may want to adjust expectations on that. LLMs deliver many things but not that. RAG etc will help, but they'll still hallucinate stuff. Especially on subjects like accounting where isn't a huge amount of training data so the base performance is quite weak
____vladrad@reddit
I have 2 and for a small company like that you can run q4 235B at like 70 tokens a second. It’s really well trained in mcp so it’ll just plug in. Dm if you have questions.
DoggoProfessor959@reddit
Would you like to try setting up helix.ml ? We have rag and a neat setup for on prem machines. If your company is below 10m annual revenue then it’s free too. Code is on github :)
MelodicRecognition7@reddit
I'd go with a new RTX PRO 6000 and a second hand EPYC4 (12 channels of DDR5-4800) system. Do not buy 3090s as they are a previous century technology and multi-GPU setup is a pain in the ass; do not buy any DDR4 CPU (even 8 channel EPYC) as if you would ever need to offload part of the inference to the CPU the speed will be awfully slow.
Murhie@reddit
Let me know what you went with in terms of hardware setup please. Also, where in NL is this accounting firm? Can you have the LLM set up a dubbeldekker for me?
Vikfro@reddit
Is it really common for such companies to go on-premise only? Using Cloud doesn't mean privacy isn't respected.
GreenTreeAndBlueSky@reddit
I'm just sweating realising small companies are spending 10s of thousands of $ on AI and taking advice from reddit on what to do
freeagleinsky@reddit
Since you talk about trust.., whom is to be trusted here?
Kimura_4200@reddit
I think it's great, as OP said it's an experimental project. Companies that invest few thousands for their IT guy to play around with big GPUs they understand that the value that could come out of it is worth much more than the price of the hardware. Also, OP will learn a lot, Reddit is full of nerds happy to share their knowledge.
Fast-Satisfaction482@reddit
I think it's great. If they took advise from some commercial advisor, they would easily spend a multiple just getting info and getting hardly anything more solid.
AFruitShopOwner@reddit (OP)
yeah thats why I'm here haha
DistributionOk6412@reddit
welp, there's a lot of info out there and tbh running llms locally is a tough journey, get ready to go down a deep rabbit hole. i've been learning about llms for a while now and honestly it's super hard to get right, even at a small scale in prod. lots of opinions on reddit, but a bunch of them are either wrong or missing key stuff. good luck!
AFruitShopOwner@reddit (OP)
I've been going down the rabbit hole ever since I tried out OpenAI's GPT2 playground. Can't wait to actually run these things myself
DistributionOk6412@reddit
it’s a slightly different hole, but glhf
CarelessParfait8030@reddit
That's what she said.
Sorry, couldn't restrain my inner child.
CarelessParfait8030@reddit
The important thing with any advice (anon or not) is the ability to verify.
As long as you can verify what's being said it's not a problem.
The main issue with anon advice is that you may have a bandwidth problem: too much to check.
But if SO showed as anything is that with a good scoring schema it's not actually a problem in practice.
ResponsibilityIll483@reddit
Actually not wild, even for a small business. Also businesses get to write expenses like this off their taxes, so consider that a 20% discount.
private_final_static@reddit
To be fair, thats probably pennies for them
vibjelo@reddit
If OP said "Whatever you say we'll buy and use for sure" then yeah, I'd agree with you.
But OP seems to know a little already, and asking for more options, typically part of a "fan out" exploration process where you try to find as many options as possible, and then after you do a "fan in" process where you evaluate everything and chose whatever fits with your tradeoffs and context.
I don't see anything wrong with that, having a wide net of options available before you filter down what actually can work sounds reasonable to me, granted OP continues to use their head and brain even after being recommended things by reddit.
TLDR: If OP thinks and doesn't blindly go with whatever redditor says, what's the problem?
phao@reddit
Well.. to be fair, they are problably trying to get free advice from wherever possible. This subreddit is just one of the possible places.
ozzeruk82@reddit
Exactly! Nice work if you can get it.
I bet there are people going from business to business in their local area demoing RAG systems and selling a couple a week for 10k+ a pop. It's probably a really smart thing to do, instead of sitting around commenting on Reddit.
Maleficent_Age1577@reddit
If they would pay that advice they would have both expensive setup and 5k less.
Stalwart-6@reddit
Hi, ollama contributor , ai engineer here. If you are really facing difficulties and dm , get it done for nominal fee. You are making correct hardware choice, and ollama has some headache setup with variables, and keeping model in memory. You could use Vllm for higher multi user throughput . #swamix_global
inaem@reddit
Maybe start with a model, rent some GPUs on the cloud platforms to try it out, then build the system.
70B is good and all, but if a 14B can do the job, you can serve more faster.
I know Western companies are allergic to it, but try Qwen 3 with thinking for a start.
inaem@reddit
I am currently serving Qwen3 30BA3 with vLLM and manually scale between 2 3090 and 4 3090 according to demand.
PutMyDickOnYourHead@reddit
The Nvidia Tesla models are going to be more expensive but more suited for enterprise workloads. They'll require some specialized cooling though (it's not hard once you find the right fan parts).
What you should buy depends on the size you're trying to run. If you're going with a 32Bish model, I'd go with 1x or 2x Nvidia A100 80GB to run enough parallel requests quickly. Even a 70B model would run fast on 2x A100 with a decent context window for multiple requests.
If you haven't done so yet, I would launch the entire service on a cheap GPU hosting service to see if the program meets your needs. You can rent them for $1/hr, so you can do a pilot with the staff to see if they find it useful before throwing serious money at the project.
Lissanro@reddit
A lot depending on a model you plan to run. Without that, it is hard to advice on hardware. With your budget, I think getting EPYC platform with 12-channel DDR5 with 768GB RAM + Pro 6000 would be the best use of money.
You can then use ik_llama.cpp for fast R1 671B inference using IQ4 quant, for example (it can work as - with Pro 6000 it will be possible to hold over 100K context in VRAM and with its bandwidth it is reasonable to expect 300+ tokens/s for prompt processing (based on the fact I am getting around 150 tokens/s with 4x3090 GPUs without tensor parallelism). Since I am getting around 8 tokens/s with old single-socket EPYC 7763 with 8-channel DDR4 + 4x3090 GPUs, with 12-channel DDR5 + Pro 6000 I think it would reasonable to expect around 20 tokens/s generation speed, but obviously this need to be tested. I shared how to setup and run ik_llama.cpp , in case you would like to give it a try. Here I documented how to create a good quality GGUF also.
Another alternative, buy cheaper DDR4-based EPYC and a pair of Pro 6000 instead, then you can load good quant of Qwen3 235B entirely in fast VRAM and it will be much faster, but quality will not be as good. I use mostly R1 exactly for this reason, but a lot depends on what kind of agent workflows you have in mind. Obviously, if faster 235B model is good enough for your use cases, then there is no need to use slower 671B.
Of course, if manage to get 12-channel DDR5-based system with a pair of Pro 6000 (it would be close to the upper bound of your budget), you get even more versatile machine where you can run fast 235B in VRAM, and 671B model in GPU+CPU mode still at decent speed when you need more intelligence. Different tasks can work better with different models, so not necessary to run the heaviest models all the time - for example, for summarization of not too long documents, much faster and smaller models can work even better.
AFruitShopOwner@reddit (OP)
Thank you, this is so helpfull
KernQ@reddit
This is an interesting option as you gain the potential of R1 Q8 for slow (overnight?) batches. 10t/s is a realistic goal, which for async interactions in an accounting firm might be usable (as you said, an email with "here are your results").
I have a 9355p 768gb DDR5 workstation. If you go this route I suggest active cooling on the RAM from day 1 (or put case fans on max as in a datacenter). Running R1 on them gets hot quickly - in the danger zone for damaging them. Fans directly blowing on them top-down will be enough.
My best piece of advice, which I know you won't listen to 😆 would be not to run before you can walk. You don't know enough about what you need. I know it's boring and new kit is exciting, but all the RAG, models and inferencing engines, software etc can be tested and validated for your domain without pulling the trigger on hardware. Your glue-layer, which I assume you're writing, can be run against hosted models using the same APIs as you'll use locally (essentially). You can deploy AWS or Google Cloud instances that simulate your planned GPUs. You don't know if vllm and tensor parallelism is something that works for your needs, or if MIG might be appropriate for your models, and so on.
That said, high RAM EPYC rigs are sweet af! I don't think you can go too wrong with them (stay away from dual socket). If it doesn't end up being used on this project it makes an amazing VM host (or workstation!). And if you splurge on 2xRTX6000 and it doesn't work out you'll be able to resell them. Ping me if you have any specefic build questions 🤓
AFruitShopOwner@reddit (OP)
Thank you so much :)
No_Conversation9561@reddit
maybe tinygrad has some offerings
stefan_evm@reddit
good idea! they do. the tinybox red for 15k$. https://tinygrad.org/#tinybox
but: as far as I know, you have to import this to Europe by yourself. No chance to buy at a European wholesaler.
stefan_evm@reddit
My 2 cents.
1) Cloud vs local: ideas for justification regarding information security.
Some claim that only cloud-based solutions are secure, and that local hosting is inherently risky. That is simply not true. Anyone making such claims should do their architectural homework – and ideally consult with their IT security officer. So should you.
Ideas:
User data, e.g. vector database for RAG etc., can and should be in your "normal" secure IT infrastructure/environment (simple linux machines or so). You "only" local host AI compute power.
2) Hardware
A wild idea of mine: Used or refurbished Mac Studios with an M1 Ultra and 128 GB of RAM cost around $2,500. You can allocate up to 125 GB of that RAM as VRAM, which is enough to run something like Qwen 235B in Q3 — or other MoEs, for that matter. Qwen 235B at around 18-20 tokens per second prompt processing. And huge contexts possible thanks to sliding window attention. Q3 really is the hard limit — but it works.
So theoretically, you could source 10 of these Macs from the secondary market and set up appropriate load balancing (see the GitHub page of llama.cpp — there are instructions for this). That way, at least 10 people could use the LLMs in parallel with good tokens-per-second rates. However, I wouldn’t allow more than two users per machine at once, otherwise performance drops too much. So that makes a maximum of 20 users in parallel. The only downside: prompt processing. Depending on the model. Especially smaller ones (Qwen 30b or Mistral 3.1) you’re limited to around 300–600 tokens per second. Larger models are slower.
You could mix large and smaller models and load balancing accordingly etc.
And you could start buying just one. And experiment.
Eden1506@reddit
Short answer for running qwen 235b: M3 Ultra 256gb or Thread-ripper with 6 x RTX 3090 Price 7-8k
For coding assistance devstral 24b is your best bet as when it comes to code assistance it can hold its own against even the large state of the Art models but only in coding obviously.
As for hardware here are some example of someone running qwen 235b A22b IQ3 at 25 tokens/s with 5x3090 on an older epic platform with ddr4.
https://www.reddit.com/r/LocalLLaMA/comments/1kg9x4d/running_qwen3235ba22b_and_llama_4_maverick/
Or another build with used parts at 5k https://www.reddit.com/r/LocalLLaMA/comments/1g6ixae/6x_gpu_build_4x_rtx_3090_and_2x_mi60_epyc_7002/
Qwen 235b at IQ4XS is comparable to Q4ks and only 125 gb in size. Meaning with 6 RTX 3090 (6x24 =144 gb ) you should be able to run it with space for context to spare and at around 20-24 tokens/s or roughly 15 words per second.
Any threadripper with 256gb + 6 rtx 3090 should do.
Alternatively you can get a Refurbished Mac Studio Apple M3 Ultra Chip 256gb from apple for 6-8k with some luck
At 800 gb/s bandwidth its not far behind the RTX 3090s 936gb/s but its actually bottleneck by apples gpu in this case
Here are some performance numbers through most are in 8 bit so you can actually expect more in lower qwant
https://www.reddit.com/r/LocalLLaMA/comments/1kfi8xh/benchmark_quickanddirty_test_of_5_models_on_a_mac/
AFruitShopOwner@reddit (OP)
Thanks, what do you think about building a system around one or two pro 6000s?
Eden1506@reddit
Nearly forgot one aspect:
For interference software vLLM has the best multiuser interference support for actual asynchronous inference as far as I remember but you will have to use atleast fp8 I believe. Rtx 6000 pro does have fp8 support but consumer cards do not and will be much slower using that format.
Meaning that with consumer cards you will not have parallel interference but instead one has to wait for the last request to finish.
That is old knowledge and might be different now so correct me if I am wrong.
AFruitShopOwner@reddit (OP)
Thanks! I'll keep it in mind, havent come across this before
Eden1506@reddit
Just did some research, using huggingface TGI its possible to use asynchronous interference with gptq q4 models afterall through I cannot say how good or fast that would run.
Eden1506@reddit
If you have the budget sure:
https://www.reddit.com/r/LocalLLaMA/s/pW8aL1x5Q9
Considering that qwen235b has 22b active parameters you should get similar performance to what mistral small 24 b gets in the example above.
Meaning around 50 tokens/s at Q8 and around 80 tokens/s in q4.
Secure_Reflection409@reddit
The 3090 is the answer to everything here :D
The only sane answer is as many Pro 6000s as you can get budget for.
fabkosta@reddit
I have seen this type of request before - and it's usually flawed from the ground up.
Think like this:
Apparently, cloud is insecure. I get that. So, what is their security concept regarding keeping the physical hardware alive that they need to operate? Are they going to rend a rack in a data center? If yes, do they trust the data center to know what it's doing? How about flooding (this is the Netherlands!), can they survive a loss of all data in one physical data center, or do they need mirroring to a second data center? If they need mirroring, are they prepared to run two or more racks physically mirrored in separate locations?
And can they do everything for 10k to 25k investment?
I hope you start seeing why I am saying the entire approach is flawed. Sure, you CAN do things yourself, but that requires MORE planning and security than purchasing cloud services. With the given budget it seems to me your company is completely oblivious of the dangers it has to NOT use the cloud but do it themselves.
Often times, small companies only see the danger lurking in one corner and conclude, they should move to another corner, remaining completely oblivious that their new position is even more dangerous than the previous one.
If you want to go down this path, make sure you do a proper comparison of all options. I would recommend using an FMEA or some similar framework for that, to ensure you are systematic about the pros and cons of each approach chosen.
By the way, don't kow abou tthe Netherlands, but at least in Switzerland there are already first cloud providers offering completely Swiss-local cloud setups, guaranteed to run only and exclusively in Switzerland. I would assume something similar to exist also in the Netherlands.
After-Cell@reddit
So… Runpod and GDPR can chill, And migrate to local if things start to make sense ?
Maleficent_Age1577@reddit
Thats not a problem. You build easily a setup with 25k. Then you add few hdds for backup data which arent in same place as companys intranet is. That simple.
AFruitShopOwner@reddit (OP)
yeah our company already has all of this infrastructure. This new system would just be for inference
laurentbourrelly@reddit
Base model Mac Mini M4 plus external SSD works fine with the Local RAG I’m selling to lawyers.
It would work fine for accounting firms. DM me if interested to talk about it.
Maleficent_Age1577@reddit
slowslowslow.
762mm_Labradors@reddit
Mac's a great to begin learning AI/individual use, I wouldn't roll one out to production for a whole company to use. Not to mention if your needs change and you want to start playing with video/photo generation.
laurentbourrelly@reddit
Nothing beats a rack of GPU connected via PCIe.
That's for ML/LLM fine-tuning, etc.
For a Local RAG, no need for super hero hardware.
Maleficent_Age1577@reddit
exactly. mac is ok for one person use for questions and thats about it. same goes a pc without gpu.
never for company that has more than one worker.
laurentbourrelly@reddit
No it's not.
Once documents are embedded and vectorized, it's all good. If you want to go PC, an RTX 3070 is plenty.
If you give it tons of documents, a base Mac Mini will be slower than a Mac Studio, but that's the only downside.
dangost_@reddit
We got A6000 in our office and running llama:3.3 70b pretty well. But It can’t handle big production. One GPU can only be used for some devs reasons
Kimura_4200@reddit
With 48GB, isn't it better to run a 32b model with full context ?
Swoopley@reddit
Hey,
Quick overview of what I went with:
Threadripper Pro 7965wx, 256gb DDR5 6000 kit of 8, Asus Pro WS WRX90E-SAGE SE, 4x 1tb t700, Silverstone RM44 case with FHS 120x fans.
As for the GPU I went with a L40s, basically a rtx6000 but passively cooled as those werent available anywhere.
still 48gb so works well with one of the fans funneled into it.
Also have a 4060ti 16gb dedicated for image/audio ai stuff and embedding.
Most of this was purchased through Azerty since we have a contract with them, totaling this setup to about 14k.
With a lot of ease into more cards when needed.
It has been running fine without issue for almost a year now. Sure it won't run everything but it will do a lot that can be considered capable. 70b models on a single card is quite nice especially with unsloth finetuning on our own data.
That's that hardware side, on the software side I'm running native ubuntu with everything in docker.
Don't worry about all those discouraging you on doing your own research, I do it the same way myself on here as well. That's how I managed to get my company to buy this hardware, had I not put in the effort of researching everything myself making myself experience in the matters they would have never greenlit it.
Oh and I use OpenWebui as the primary front-end for my colleagues, due to how simple it is for the end user, not to complicated and such like other front ends yet having a lot of features.
As for other software that integrates with it like OCR and such, you're better off researching that separately from the hardware since that is mostly done on CPU so no worries there. If you were to setup RAG with all your documents then it would be nice to have a dedicated GPU but that is not really needed per say.
Some embedding models work fine on CPU, it's just the main putting everything into the datastore getting it all vectorized that you'd prefer to work on a GPU but that's a one off.
For engine go with your flavour, Ollama is good enough for 5 active concurrent users but I wouldn't go futher than that. Simply vllm or sglang or aphrodite or llamacpp is all you need.
Hope that helped
Skrachen@reddit
How many people are you serving with this setup ?
Swoopley@reddit
give or take 20 active users of which 5 heavy with lots of context documents.
AFruitShopOwner@reddit (OP)
Thanks, this is really helpful!
disillusioned_okapi@reddit
something that we rarely talk about in this subreddit that in my professional opinion will be very important in your case is reliability / high-availability.
If you do decide to go this route, and if your org adopts their processes to use this new infrastructure, you really don't want this system to go down.
And for that specific concern a single PC build, no matter how many GPUs it has, will not be something I'd recommend.
If I was in your place, I'd get 3 beefy Mac Studios with M3 Ultra, and setup a vllm cluster. That way if one machine goes out of service, you have performance degradation and not a complete downtime.
I've already been seeing posts like these, where people have created pretty decent infrastructure-as-code solutions as a good starting point.
To be clear, this won't be the most performant setup for the money, but it'll be a whole lot more reliable than a single machine, especially if this infra ends up being business critical.
gaijingreg@reddit
Now this is architecting.
AFruitShopOwner@reddit (OP)
Reliability/uptime is really not much of a concern at all. I actually don't want my colleagues to depend on AI for their work. At least not for the next couple of years.
disillusioned_okapi@reddit
in that case please discard my comments. if reliability isn't desired, then the performance loss due network bandwidth would make this setup a waste of money.
MorDrCre@reddit
What experience do you have with setting up the security on what is going to effectively be a private cloud? Are you going to have off-site access? Is your local LLM only going to access locally hosted docs (I guess on other local servers), or also checking with national tax authority/government sources? How are you going to make sure that you won't have any holes through which intruders can pass? I see in another reply you've mentioned OP that the company has preexisting infrastructure...how resilient and fault tolerant is it? What's the worst case scenario that the partners/owners are prepared to tolerate?
AFruitShopOwner@reddit (OP)
my dude I'm just building a single inferene machine, all of the other IT is handeld by our capable IT department
MorDrCre@reddit
Ok guv 👍 , that sounds better for you 😉
As long as you can put the responsibility for the security on to the IT department, you'll probably not too badly off. But do what you can to limit your responsibilities to setting it up and maybe the initial configuration - IT has responsibility for hardening, dealing with all network issues etc (and get it on paper).
RAG is frequently Agentic RAG nowadays and prompt injection is a thing, so avoid the "lethal trifecta" of Simon Willison (https://simonw.substack.com/p/the-lethal-trifecta-for-ai-agents)
AFruitShopOwner@reddit (OP)
Thanks! Very helpful
GodIsAWomaniser@reddit
Sounds like you need to hire someone to do risk management with domain knowledge regarding AI. I'm available with 2 weeks notice, DM me your organisation's email.
Slight_Antelope3099@reddit
You can do it, but it’s not cost effective for the same performance. People doing it are usually hobbyists who don’t wanna pay 1-2k for a gpu and are fine with speeds of <5 tokens per second and very small models, I doubt that’s what ur company wants
My comment was still kinda unnecessary I guess sorry
cnmoro@reddit
If you want to serve multiple people at the same time you should use vLLM as inference engine
AFruitShopOwner@reddit (OP)
I'll check it out, thanks
tcpjack@reddit
CPU is slow. I've been running a single 3090+768gb ddr5 5600. Recently picked up rtx 6000 pro and try and stay on gpu as much as possible. Especially with reasoning.
I'd recommend a single cpu+rtx pro 6000. 384gb ddr5. 2x4tb raid1 ssd disks for system, and 2x2tb raid1 nvme for your models.
Could go with more system ram, but a second rtx pro 6000 would go much further than adding more ram/cpus.
AFruitShopOwner@reddit (OP)
Thanks this is exactly the info I need
BenniB99@reddit
Just to maybe answer your questions more directly:
Hardware:
Definitely GPU-based, CPU centered systems might work okayish for smaller or MoE models but will never scale well (especially in multi-user scenarios). Just make sure the surrounding hardware platform around your GPU(s) supports sufficient PCIE-Lanes (if you are going for a multi-gpu setup or plan to add more in the future).
This is a bit of a trade-off. Consumer GPU(s) will be available much cheaper (especially used) than a single workstation GPU. For instance 4 3090s might cost about 700-800€ a piece and will result in the same amount of total VRAM (96GB) as a single RTX 6000 Pro for 7-10k €.
However there are two things that should be considered here:
So generally, if money isn't that big of an issue, I would prefer a single card with a lot of VRAM as opposed to multiple gpus which have the same amount of VRAM when combined. This also makes adding more cards much easier later on if necessary. :)
Software:
One hundred percent, it is much more performant than i.e. Windows and Framework support is much better (especially for training).
If you offload part of it to system RAM you will be able to run it with reasonable speed since it is a MoE model.
However this will not work well in a scenario with potential multiple requests in parallel.
You would need around 140GB VRAM to run just the model fully on the GPU in 4bit.
In general you should keep in mind how the large the context might get for your use-cases as this will need a lot of additional VRAM on top of the model itself.
Well the easiest to set up would likely be ollama (although I do not consider this the most user-friendly option personally). I think llama.cpp already has all one might need to get started quickly i.e. a CLI, a python wrapper or a simple openai api compatible webserver so it can easily be integrated into existing frameworks or tools.
For high-performance inference you will likely want to take a look at vllm or sglang.
Vector databases and RAG frameworks are a dime a dozen. I think it usually pays off to build your own RAG system.
AFruitShopOwner@reddit (OP)
Thanks, very helpfull
Slight_Antelope3099@reddit
If u have to ask whether to use cpus or gpus ur neither able to build the hardware nor the software side
AFruitShopOwner@reddit (OP)
you know you can build an AI inference system without GPU's right? Sure its slow to do it all on the CPU but it works and its pretty cost effective
Pixer---@reddit
Maybe wait until intel releases their arc b60 dual cards with 48gb each for 1000 € and get 4-8 of those in a system. The easy route is a Mac Studio with a m3 ultra and 512gb ram, which gets you 20 t/s on r1 671b q2
AFruitShopOwner@reddit (OP)
Yes I was considering waiting on those too. I'll probably end up going with Nvidia anyway because CUDA support is a big deal
Saschabrix@reddit
AWS is your friend.
AFruitShopOwner@reddit (OP)
not an option sadly, boss wants it local
yachty66@reddit
Hey. I've been building hardware for local models, see here https://x.com/yachty66/status/1928490563961016726 - and we are also planning to build a bigger system. If you want, reach out to me; I am happy to answer questions.
AFruitShopOwner@reddit (OP)
Thanks, I might take you up on that later
nonerequired_@reddit
Happy cake day btw
AFruitShopOwner@reddit (OP)
:) ty
vibjelo@reddit
How are you getting 4x 3090 for sub-$1000, assuming the non-GPU hardware costs at least $1000? Is it all second-hand hardware?
yachty66@reddit
I got them on eBay, second-hand hardware, yes
kkingsbe@reddit
Open web ui + ollama. Super simple and can run on a MacBook
You_Wen_AzzHu@reddit
Renting a VM with multiple GPUs from Azure is more cost effective.
Past-Grapefruit488@reddit
Start small. Just get a Mac Studio to host LLM and a Linux server to run rest of stack (search, CMS, Agents)
HorizonIQ_MM@reddit
It’s great that you’re thinking through your AI strategy before jumping into hardware. Too many people start buying GPUs before defining their workloads. Understanding your model size, use cases (RAG, summarization, agent workflows), and performance expectations will make a big difference in choosing the right infrastructure. Since you're focused on privacy, latency, and in-house control, that early planning is going to save you a lot of time and money.
If you decide not to build the system yourself, HorizonIQ provides dedicated GPU and CPU servers on an IaaS model. You can rent month-to-month to get started, there’s no shared infrastructure, fully isolated, and customizable to your specs. For teams prioritizing data privacy (like your accounting use case), this gives you performance and control without putting anything in the public cloud.
Many of our customers are in a similar situation—building these Local AI systems to avoid outrageous token fees. We also have data center space in the Netherlands, so if latency or data sovereignty is a concern, we can keep everything close to home. That’s especially useful if you ever scale and want to avoid managing physical hardware on-site.
Let me know if you'd like to see sample configs for your RAG and automation workloads—happy to help.
These-Lychee4623@reddit
Note that to utilize Mac to full extent - NPE, MPS and CPU, the AI models needs to converted to CoreML which is sometimes non-trivial.
Zestyclose_Ad8420@reddit
before buyng the hardware you have to understand exaclty what kind of performance you want, try different GPUs on runpod, rent baremetal servers with GPU, do the sizing, then move on to acquiring hardware.
for development you need to involve actual developers to pipe together all the things to make it work properly
ozzeruk82@reddit
Good luck, but I can't be the only one somewhat shocked that your company is giving you a budget of 10-25k and you admit that you don't actually know what you're doing. I guess you convinced them that you do know what you're doing.
Personally, with that budget, you could probably go down the Mac route, costly but quieter, and cheaper to run than a series of consumer GPUs.
Maleficent_Age1577@reddit
if that company has more than 1 guy using llms then mac is not an option.
ozzeruk82@reddit
Which is exactly why I said we need to know how much use there is.
If it's one use per 30 mins, then a Mac will be fine.
AFruitShopOwner@reddit (OP)
oh sure theres some dunning kruger going on but I do have IT experience (stuff like building computers and running linux are not at all new to me) and our company has an IT team that can assist me, they are also okay with me trying this out.
AllanSundry2020@reddit
i would spec out the use cases thoroughly. Then decide on solution where that takes you
cunseyapostle@reddit
Just use an API into Claude. Seriously, this is a business not a hobby.
PsychohistorySeldon@reddit
A number of paths you can take from a hardware perspective, but it'd be inadvisable for an accounting firm to tackle hardware directly (not because you can't, but because it's a poor use of your resources).
Alternatively, I'd recommend running all this inside a VPC in an EU instance in one of the major clouds, so it's air gapped from the rest of the world and completely private. You'll find just the maintenance of hardware and networking/availability will take too much of your time.
Since you're an accounting firm, accuracy and testing is pretty essential. I'd use a https://bem.ai or similar for document ingestion and extraction and then build agentic workflows on top. For RAG/vector storage, just go for pgVector. For production loops, golang and python are kings.
Relevant_Helicopter6@reddit
Maybe you should ask ChatGPT.
Antique-Ad1012@reddit
i dont think the hardware is not there yet at this price point to serve a 100 people reliably. You will most likely end up spending a lot more, simply because the hours that you will be put into getting something to work for your usecase.
maybe most importantly your system will be significantly less capable than online services
in my experience this requirement is the most difficult to get working reliably on a local system:
"various files like contracts, invoices, emails, excel sheets, word files and pdfs."
im located in the Netherlands my self (Eindhoven region) and experimenting with running local llms on an m2 ultra send me a dm if you would like to brianstorm a bit ;)
AFruitShopOwner@reddit (OP)
sorry I should have been more specific. I dont think more than 20 people will actually be using this AI system. Probably not more than 5 at the same time during peak load.
Maybe that will incease as the system gets better.
terminoid_@reddit
the RAG/summarizing/agents stuff can be handled by a consumer Nvidia card just fine. Coding is the most difficult of your requirements imo, and requires the most investment. i think a single pro 6000 would be the best way to go overall.
AFruitShopOwner@reddit (OP)
Thanks! I was already kinda planning to build this system around the pro 6000. Maybe score a second hand EPYC system to put it in.
Desperate-Sir-5088@reddit
How about DGX SPARC 128GB RAM + 4TB SSD @ $3,999