Planning to build AI PC does my Build make sense?
Posted by germaniiifelisarta@reddit | LocalLLaMA | View on Reddit | 13 comments
Hi so I've been looking all around and there seems to be a shortage of GPU guides when building a PC for AI Inference, the only viable reference I could consult are GPU benchmarks and build posts from here.
So I'm planning to build an AI "Box". Based on my research the best consumer-level GPUs that are bang for the buck would be the RTX xx90 24GB series. So I browsed my local marketplace and those things are so dang expensive. So I looked for an alternative and found the RTX xx60 16GB line. Which has lesser vRAM but more in my price range.
I also found that I could cluster (not sure if this is the correct word but something something SLI) GPUs.
I was aiming to use the AI box purely for inferencing so I would be loading up LLMs, VLMs and try Stable Diffusion not at the same time though.
Sooo, based on those above, I have a few questions:
-
Is the RTX xx60 non/Ti 16GB models have acceptable performance on my use case?
-
If not, is it possible to do the clustering if I would buy 2 RTX xx60 non/Ti 16GB?
-
Am I making sense?
All help is appreciated. Thanks if you think there is a better sub, please let me know and I would ask there too
remghoost7@reddit
I'm just going to ramble a bit in the direction of your post, offering some information that might help you on your way.
This is no way a "definitive" answer, but more recommendations based on information what I've learned over the past few years.
Obviously, all AI rigs serve different purposes depending on the models you want to run / functions you want to use it for / etc.
For reference, I've been knee-deep in the locally hosted AI space since late 2022.
With the disclaimers out of the way, I'll start with my own setup, which is sort of a "Ship of Theseus" build that I use to for AI (LLMs and image/video generation) / software development / playing games:
This setup can run most things that I throw at it, which was my goal.
I wanted a "jack of all trades, master of none" sort of build.
As mentioned, I use it for software development and games as well.
My next upgrade will probably be an AM5 board (with more PCIe 16x slots), CPU, and probably 128GB of DDR5 6400Mhz.
My LLM of choice recently has been the unsloth Q4_K_M quant of GLM-4-32b-0414, which can run entirely in VRAM with a 16k context.
I get around
24t/sfor generation on a 4k token context.And my SD model of choice has been Chroma-v38-detail-calibrated running at fp8 (converted using this script).
I can generate a 1024x1408 image (using sageattention/magcache/torch.compile) in around 35 seconds. 26 steps @
1.35s/it.While you can run cards in "SLI", there have always been quirks/hangups.
When it comes to LLMs, it's not really a problem. Offloading (either to system RAM or secondary cards) has been present since day 1.
Where it gets a bit tricky is with image/video generation. Stable Diffusion models are not a big fan of offloading.
Granted, it's way better today than it was a year ago (and will continue to get better), but keep that in mind.
Many people run 4x3090's and have no issues.
Locally hosted AI is a slippery slope.
A very "if you give a mouse a cookie" sort of vibe.
You will want more VRAM. Always. Forever.
And PCIe lanes/slots are arguably worth more than the cards that are in them.
My primary bottleneck at the moment is how many cards I can literally fit on my board at full speed.
I plan on grabbing another 3090 down the line (I eventually want 4 of them), but if I slap another one into my board right now, they'll both run at 8x speeds (because my board is meh). Hence my next upgrade being an AM5 board. And even then, I'm more considering either a Threadripper or dual Xeons for the PCIe lanes.
I won't recommend any specific hardware, but I'd recommend you consider upgrade potential in the future.
If you eventually decide that you want more VRAM (which you will), having two lower VRAM cards will force you to ditch one of them (due to the PCIe lanes/slots issue mentioned above).
Save up for the xx90 class card. I bought my used 3090 for around $900-ish.
They were $500 back at the end of 2024, but people are gobbling them up like hotcakes nowadays.
Expect to pay at least $1000.
But I freaking love this card and there are no regrets on my purchase.
I will be buying more of them in the future.
But yeah, those are just my two cents.
Take all information with a grain of salt.
germaniiifelisarta@reddit (OP)
With regards to the SLI, It would be possible to just utilize one GPU for it right? In the scenario that I would have let's say 2 GPUs I could use 2 GPUs for LLMs, then easily switch back to just 1 GPU for SD?
remghoost7@reddit
Absolutely.
That's essentially what I plan on doing.
I have an old 1060 6GB that I might repurpose for this use-case.
It could handle a decently sized LLM with CPU offloading.
MOE models are neat for this sort of thing as well, since they usually have a pretty small active parameter count.
I sort of want to try to mod it with more VRAM as well though, so that could be neat...
But yeah, you could have one card that deals with just image/video generation (since those models prefer to be on a single card) and the other one could handle LLMs/TTS/etc.
In response to your other comment:
A lot of people don't.
Most people just try to grab the the largest VRAM cards they can find and don't consider the "supporting cast".
It's not really an issue when you have one card, but PCIe lanes/slots fill up super quick.
Don't forget, an M.2 drive usually takes 4 lanes as well.
My CPU (Ryzen 9 5950x), which is the "best" CPU for the AM4 socket, only has 20 Gen 4 PCIe lanes.
Hence why I mentioned Threadrippers / dual Xeons. For reference, a Threadripper PRO 7000WX has 128 Gen 5 PCIe lanes.
But a setup like that can cost upwards of $5k, not factoring in the GPUs.
And you can't really game on those either. They have obnoxiously high core counts, but extremely low clock speeds (usually in the 2GHz range).
Hence why I'm leaning more towards an AM5 socket/motherboard.
I could technically add another 3090 to my setup, but they'd both be running at 8x speed.
It shouldn't be too much of an issue for inference (since a 3090 should barely saturate an 8x PCIe Gen 4 connection, from my research), but it'd take a noticeable hit on training. This doesn't really become a problem if you have Gen 5 lanes/slots, since an 8x Gen 5 slot has the equivalent bandwidth of a 16x Gen 4 slot.
But yeah, the motherboard/CPU rabbit hole is pretty deep on this one.
I've done a bit of research, but there's still a ton I don't know about it.
germaniiifelisarta@reddit (OP)
Thanks so much for expounding!
May I know you current motherboard/build/setup and your target AM5 build? Just so I could get a reference.
Thanks!
germaniiifelisarta@reddit (OP)
Thank you very much for the very in-depth explanation.
This is what I wanted too and also since I'm already going to be using a powerful gpu, why not use it for gaming also right?
--
I understand that vRAM really is the biggest variable in this scenario and I think your advice is quite strong. I didn't take into consideration my motherboard (I only use a Gigabyte A520i AC atm, but I will definitely upgrade in the future back to ATX/mATX).
--
Also, thanks for sharing you inference speeds, I find it very useful. It gives me an idea how fast it could process things.
No-Consequence-1779@reddit
This is 1200$
CPU: AMD Ryzen Threadripper 2950X (16-core/32-thread up to 4.40GHz with 64 PCIe lanes) CPU cooler: Waith Ripper CPU air cooler (RGB) MOBO: MSI X399 Gaming pro GPU: Nvidia Quadro RTX4000 (8GB GDDR6) RAM: 128GB DDR4 Storage: Samsing 2TB NVME PSU: Cooler master 1200 watt (80+ platinum) Case: Thermaltake view 71 (4-sided tempered glass)
Add 900 per 3090 gpu. Get 1-2 to start.
BobbyL2k@reddit
Seems like you don’t know what class (size) of models you want to run. I can recommend you buy a CPU system first. Get 96GB (48GBx2) of DDR5 @ 6400MHz. Then try running all of the models, then it will be obvious how much you need to buy (or save up to).
The thing about RAM is that it’s always nice to have. Once you have a GPU, the RAM is still useful. So it’s a safe buy when you don’t have enough understanding of what you want.
And get a board with two PCI-E 5.0 slots. So you can add in two GPUs later. I’m assuming you’re not going to buy more than 2 GPUs, but if you are the same advice applies, just with a EPYC system.
germaniiifelisarta@reddit (OP)
I actually have a server box right now that's 64GB DDR4 3200. I have tried running ollama on it with ~7B models and it works okay. Not so great responses but the speed was pretty okay. If I buy a GPU, would it be the same speed? especially if for example I go the Agentic Route(multiple requests at a time)?
BobbyL2k@reddit
I forgot to answer your question about speed. Assuming the bottleneck is bandwidth, which it usually is, you can check the memory bandwidth on Techpowerup GPU database and estimate the speed (tok/s).
Maybe you see results on here with X GPU with 500GB/s of memory bandwidth getting A tok/s. Y GPU with 1TB/s of memory bandwidth will get around 2xA tok/s.
BobbyL2k@reddit
For llama.cpp (and which uses llama.cpp like ollama), if you want to process requests in parallel, the context size will be divided by the parallel config.
Imagine this, the your VRAM and RAM is used to store two things: the model and the context. Let’s say you have xGB of memory and it can fit the model + 64K of context. You can either allocate the whole 64K context to one request (the default option) or split up the 64K context for use in parallel. It might be 4 parallel requests each with 16K context, or 8 parallel requests with 8K context, and so on.
Now you have to ask yourself how many parallel requests do you want to process at once? Remember, you can still make parallel requests and let them queue up for their turn.
For GPUs setup, the bottleneck is the memory bandwidth. So you should have little degradation in tok/s as the number of parallel requests increases (to a certain point).
germaniiifelisarta@reddit (OP)
Wow thank you very much for explaining that! I was also trying to look into how that worked and you just answered it.
So now it's up to me to do the math based on this knowledge. Thank you very much!
jettoblack@reddit
What models do you need to run, do you know yet?
It’s like asking “What car should I buy? A F150 is nice but man it’s expensive, should I get a Corolla instead?” Depends on what you want to do with it… daily commute from A to B? Then Corolla for sure. Hauling plywood all day? You need the truck.
If you don’t know yet then try some online models until you have a better idea of your local needs.
germaniiifelisarta@reddit (OP)
Thank you for reminding me, I updated the post.
I thought mentioning LLMs and VLMs in general was going to be enough but I've added the specifics already on the post.
Thanks again!