Can Consumer Desktop CPUs handle 3-4 GPUs well?
Posted by pmttyji@reddit | LocalLLaMA | View on Reddit | 27 comments
Unfortunately we're(friend & me) in a Down the rabbit hole situation for sometime on buying rig. Workstation/Server setup is out of our budget. (Screw saltman for the current massive price RAM & other components situation.) And Desktop setup is OK, but we're not sure whether we could run 3-4 GPUs(Kind of Future-proof) normally with this setup. My plan is to run 300B models @ Q4 so 144GB VRAM is enough for 150 GB files.
For example, below is sample Desktop setup we're planning to get.
- Ryzen 9 9950X3D (Planning to get Ryzen 9 9950X3D2, releasing this month)
- ProArt X670E Motherboard
- Radeon PRO W7800 48GB X 3 Qty = 144GB VRAM
- 128GB DDR5 RAM
- 4TB NVMe SSD X 2
- 8TB HDD X 2
- 2000W PSU
- 360mm Liquid Cooler
- Cabinet (Full Tower)
Most Consumer desktops' maximum PCIE lanes is only 24. Here I'm talking about AMD Ryzen 9 9950X3D. Almost most recent AMD's have 24 only.
My question is will get 3X bandwidth if I use 3 GPUs? Currently I have no plan to buy 4th GPU. But still will I get 4X bandwidth if I use 4 GPUs?
For example, Radeon PRO W7800's bandwidth is 864 GB/s. so will I get 2592 GB/s(3 x 864) from 3 GPUs or what? Same question with 4 GPUs?
So we're not getting 3X/4X bandwidth, what would be the actual bandwidth during 3/4 GPUs situations.
Please share your experience. Thanks
ethertype@reddit
Your GPUs may have a 16 lane PCIe connector, but will happily negotiate down to 4 and probably down to 1 lane.
How much bandwidth you need between system and GPU ist highly dependent on the task at hand.
Look up bifurcation
pmttyji@reddit (OP)
Totally new to multi GPU thing. Currently I use laptop which has only 8GB VRAM(4060) so I'm new newbie to GPUs.
GaryDUnicorn@reddit
Check out PCIE switching. all the bandwidth, none of the mess: https://forums.servethehome.com/index.php?threads/new-chinese-pcie-switch-board-gpu-testing.52488/
Radiant-Video7257@reddit
I wouldn't run a 300b model without GPU's that have NV-link or equivalent for AMD. Try a smaller model and see how it works for your use case, they're getting much better (Qwen3.5 27b and Gemma 4 31b).
Ok_Mammoth589@reddit
Pcie is fine
Radiant-Video7257@reddit
lmao, pcie 4.0 is too slow, expect 10t/s max with a 300b model.
Annual_Award1260@reddit
Think you will have a hard time running 300B on that setup. Pci 4.0 all at x16 will be twice as fast as pci 5.0 at x4
Alot of models are still quite dependent on CPU/ram and 128GB fills up fast.
Have you considered the new intel b70?
Here’s my setup on a i9-13900k and Ill definitely be buying a threadripper setup as soon as ram prices drop
pmttyji@reddit (OP)
I don't want to go with 32 or 24 GB pieces as consumer desktops can have only 3-4 GPUs so I want to fill it with bigger GBs like 48 or 64 and above.
Saltman already ruined our plan for 1-2 years *sigh*
I have this question. Mentioned CPU Ryzen 9 9950X3D is PCIE 5.0. And Radeon PRO W7800's Bus type is PCIe 4.0 x16. Hope it's compatible for sure. Though Other commenter mentioned that 3-4 GPUs are possible, any idea how much speed/performance difference with this setup(Talking about PCIE 5.0 with PCIe 4.0 x16)?
Annual_Award1260@reddit
The b70 is only $950 which is actually a very good price for 32GB. May have some driver issues that still need to be worked out.
Pci is back compatible so your pci 4 card will run at pci 4 speeds in the pci 5 slot.
Your configuration when both x16 slots are in use will run each slot at x8 and x8. The other 8 lanes are used for m.2 drives at x4 and x4.
Pci 5.0 is 4GB/s per lane. Pci 4.0 is 2GB/sec per lane.
So at pci 4.0. 2x gpu = 16GB/sec per card. 3-4x gpu = 8GB/sec per card.
Pci 5.0 is double those speeds. But still x8 with 2 cards due to lack of pci lanes.
Ideally you want PCI 5 cards running at x16 to get 64GB/sec per card. Even a pci 4 threadripper setup will have 88 lanes and would give you 32GB/sec per card
If you are planning on running a separate model per card performance will be ok. If you are running 1 model across all the cards you will take a huge performance hit
pmttyji@reddit (OP)
Thanks for the detailed reply. But unfortunately it was the only affordable 48GB card. I don't want to go for 32/24 GB cards. I want to fill this setup with 200GB VRAM in future so going with high GB cards now.
With 48GB cards, I can run 70B models in single card.
We had such plan, but that setup goes out of our budget. Crappy sellers from my country selling overpriced RAM at too overpriced range. So we dropped that plan.
Annual_Award1260@reddit
There will be much larger/cheaper cards in the near future. Also I'm not familiar with the amd cards with no cuda support. I suggest you rent a few gpus on vast.ai and see if they meet your requirements. Even with my dual blackwell 6000 pro set up I still would like it to perform better.
pmttyji@reddit (OP)
I don't think so dude *sigh* It'll take at least a year. Same with RAM.
Yep, it's tradeoff on my side. Right now I need more VRAM to run big like 200-250B models @ Q4.
Also I don't want any online stuff. I prefer 100% local, at least for now.
Annual_Award1260@reddit
Sam Altman sent "letters of intent" to Samsung and Hynix locking up 40% of the ram supply, just to walk away from the non-contractual deal 3 months later. The openai funding has also dried up. Add that to google's turboquant announcement I think we will see ram prices drop very quickly.
Local has its place, but compared to frontier datacenter models like claude opus 4.6 the local models are no where close.
Farmadupe@reddit
You'll almost definitely want to design around 4 gpus * llama.cpp is way slower for multi gpu than vllm or sglang * By the time you're spending a 5 figure sum (or almost) llama.cpp probably isn't at the right level of quality. None of the stacks are bulletproof but vllm is way closer to production quality than llama Cpp * As you said yourself your planned models won't fit in vram. 144G is smaller than 150G * You'll also need overhead for kv cache and assorted compute buffers. For 150G weights, 192G vram might be a starting minimum. * However llama.cpp wants are way better than ones available for vllm or sglang
For the amount of money, I'd recommend playing around on runpod or vast.ai to set up the stackm when you've worked that out you can set it up on progressivele smaller hardware until you've found you minimum. Then you can go and buy without risk
pmttyji@reddit (OP)
You spotted those 2 numbers well. Right, Q4's size usually Model's B divided by 2. 300/2 = 150.
But for big/large models, I won't be using bigger Q4 quants like Q4_K_M or Q4_K_XL. I might pick smaller Q4 quants like IQ4_XS or IQ4_NL. Plus additionally I have 128GB RAM which's useful to manage 100K Context & Q8 KVCache. Recently we got stuff like TurboQuant, hope it brings some magic on this.
Friend is splitting the bill with me on this as he's gonna use the rig for Video Editing, Graphic/Animation related stuff.
Thanks for the detailed response.
Farmadupe@reddit
Saying bluntly just to make sure the advice across clearly: * Storing entire kv cache in system ram is an extremely bad idea. Your inference speed will be massively bottlenecked by pcie bandwidth. This will massively slow down your 300b@q4 llm to the point of being unusable. * Offloading any portion of the model weights to system ram (even a small portion, even if MoE) is a moderately bad idea. your inference speed will be massively limited by computer bandwidth. * I'm 99% sure that sglang or vllm will refuse to start with your intended to configuration. The reason that they do not support the configuration is because it is an extremely bad idea. * Llama.cpp probably will start but due to the limitations of pcie bandwidth, it will be unusably slow. * With llama.cpp layer splitting you will not get the benefits of increased vram bandwidth. Depending on specific details, your system will perform inference at either system ram speed, cpu speed (imo most likely in my experience based on my own 9950x with llama.cpp) or pcie bus speed)
Given the amount of money you're spending: * your setup is incompatible with the inference engines you should be targeting (vllm and sglang) * It will probably work with llama.cpp at very slow speed. * You will not utilize a meaningful amount of vram bandwidth.
If your hard requirement is a specific 300b model at q4, you would need to buy that 4th card to get good inference speed.
Otherwise you will need to use an alternative model.
Other redditors will have more experience than me with running 4 gpus on desktop platforms. pcie bandwidth will be a major concern.
pmttyji@reddit (OP)
Thanks for the detailed response again.
We're planning to grab 9950X3D2 Dual Edition which comes with L3 Cache 192MB (Total 208MB). Also it has AVX-512 which is useful for both llama.cpp & ik_llama.cpp.
I strongly agree with you here. Probably 2026's upcoming models gonna force me to buy 4th GPU ASAP. Currently I don't see any important 300B size models(Qwen3.5 actually has 397B, that's 400B range). For now I'll be sticking with \~250B models like MiniMax-M2.5, Qwen3-235B, etc.,
Yep, 1-2 commenters mentioned that.
Yep, I'm aware of that. We both have separate laptops already. And we both need this desktop on different days only. We rented a room for freelance work near to our home. He usually needs during weekends, as he's busy on weekdays doing his work(Graphics designer/Video Editor) at client locations. Travelling daily. So I'll be using desktop almost all weekdays day time. He'll use after evening till late night. So that's fine, no conflict.
hurdurdur7@reddit
From what i understand by looking at the card specs and current prices of all of this ... how convinced are you that you will get something that is significantly better than an m5 ultra based mac studio that is supposed to come out some time soon? It will not have any of the 3 bus pci-e overhead that you will be fighting with. (And just stating this once more - i am not an apple fanboy, i despise their software stack, but damn those M* chips are good).
As for running 300B models at Q4 quant ... i think you forgot about the size of context in you calculations. Big models also come with big context memory cost. And to my knowledge splitting the models across 3 cards due to layer sizes won't really work like this either.
Do more research, prove me wrong, i would be happy to learn too.
pmttyji@reddit (OP)
Friend & me sharing the rig, he's gonna use for Video Editing, Graphics/Animation related stuff. Mac won't be suitable for this case as some of Apps(His paid softwares like 3Ds Max) not supported yet.
Maybe next year, I'll try to grab M5 Studio Ultra 512GB/1TB variant(Good for portability). This year those variants are unlikely I guess.
Replied to other comment on this. Additional 128GB RAM after VRAM could help.
hurdurdur7@reddit
Well llama will start to push part of the model out of the gpu in that scenario, things will get slow ... then again, i think many 120B models are very usable as well at higher quants and they will fit your setup fine.
pmttyji@reddit (OP)
Agree. Still IQ4_XS or IQ4_NL could put entire model in to GPU. So Context & KVCache will be in RAM. But I hope llama.cpp's future optimizations will bring additional boost so it won't be slow in future.
Absolutely. 20-120B models gonna be my daily drivers.
Lissanro@reddit
My previous rig was based on Ryzen 9 5950X CPU with 128 GB RAM and it could handle four 3090 GPUs just fine, in x8/x8/x4/x1 configuration. The x1 GPU was most annoying since kills tensor parallelism performance and also had slower loading times. For typical llama.cpp inference it worked just fine, even though with some performance loss.
I however strongly recommend to get EPYC-based rig instead. This is what I ended up migrating to in the beginning of the previous year. Also, server DDR4 memory is cheaper than desktop DDR5 but much faster. This is because EPYC has 8 memory channels instead of two. If you plan GPU-only inference than you do not need to get the fastest CPU and memory, which can save some money.
pmttyji@reddit (OP)
OH MY .... This is the reply I wanted to see. Thanks. Hope Ryzen 9 9950X3D (or Ryzen 9 9950X3D2) is better pick by us on this setup. This CPU ticks 1] Integrated Graphics 2] AVX-512 3] PCIe 5.0 4] Max 256GB RAM
So 3 GPUs would do better, right?
Did you check bandwidth during 1 GPU, 2 GPUs, 3 GPUs & 4 GPUs? What was the difference? That's the part I want to know. (I remember that filling all RAM slots usually brings down RAM's memory bandwidth, so just wanted know how it works on GPU side)
In 4 GPUs situation, I'll move the weaker GPU to that spot. Next year probably.
Previously we planned to go with AMD Ryzen Threadripper 9960X which has 4 RAM channels & 48 PCIE lanes. Unfortunately RAM(ECC RDIMM) alone ruining the plan. Too costly now. Plus my country's shitty sellers overselling the already overpriced RAMs. Here they're selling 128GB RAM(ECC RDIMM) at ridiculously more than $5K. Also same with DDR4 RAM(Too risky to buy that at high cost). Not kidding. So no way of going with Workstation/Server route.
Lissanro@reddit
If you want to go building the rig based on the gaming motherboard, worth checking if it supports bifurcating its slots. Even a single PCI-E 4.0 x16 would work better if bifurcated to x4/x4/x4/x4 than x8/x8/x4/x1 I had, where x1 was the performance killer. Especially true if running something like vLLM with a model fully in VRAM, it is critical to have good bandwidth; for llama.cpp, it is not as important, it also will work fine with odd number of GPUs like 3 (as opposed to vLLM that would only like 2, 4, 8, and so on).
For server build that mostly focuses on GPU-only inference, DDR5 does not really make sense given the current prices, only DDR4. It literally will not make any difference on performance so only get DDR5 if you plan CPU+GPU inference. This is true both for desktop and server platforms.
If you cannot find good deals for used server parts, getting DDR4-based EPYC can be an issue, since it would not make sense to buy DDR4 as new parts (by the way, Threadripper is best to be avoided, EPYC is better AI-related workloads since it has more memory channels). I find it mind boggling how prices skyrocketed though, I got lucky and got 1 TB of 3200 MHz DDR4 RAM for about $1600 in the previous year, now the same memory is many times more expensive. Based on what you describe, sounds like even finding 128 GB of server DDR4 memory can be difficult in your case. In this situation, going with the gaming motherboard makes sense.
pmttyji@reddit (OP)
That's really great info. you shared here. I'll check for that sure.
That's really good to know.
Multiple problems in my country.
1] The crappy sellers won't price down old items. They keep same price as it as. And overprice during shortage situation. For example, they're selling 3090 at almost 4090's price because both are 24GB VRAM. Half of the innocent/fool gamers getting trapped by this situation as they're not aware of specs of those components.
2] RAMpocalypse is global issue. But in my country, crappy sellers made it as RAMpocalypse^(2) unfortunately. Too overpriced, it doesn't matter DDR5 or DDR4. It's totally useless to buy DDR4 at very high cost here.
3] ebay closed their regional website before lockdown itself. Otherwise we could find cheap or normal deals there.
4] In my country, Custom fees is huge like 38%. Rarely it goes up.
You predicted it correctly. I'll check for suitable motherboards locally.
aafirr@reddit
There is no such calculation as 3x864 because that would mean all gpus access each others' vram like internally, so you are probably stuck with 864GB/s with 144GB vram, which is actually great I think. One thing im concerned is these are amd cards so it wouldn't be comfortable to run anything, most things based on CUDA so probably be painful. Next option I would pick is trying to scavenge used 3-4 64gb-128gb mac studios.
pmttyji@reddit (OP)
Oh I see. I though it would accumulate like how RAM's memory bandwidth increases after filling additional RAM channel.
I get it what you're saying. But right now I need more VRAM. Previously we planned to get 2 X NVIDIA RTX Pro 4000 Blackwell, but it was total 48GB VRAM only. But with AMD Radeon cards, I get 96GB VRAM which's good to run 100B models better.
Next year probably if they release 512GB/1TB variant of M5 Ultra.