Setup with Nvidia 6000 Pro
Posted by Appropriate-Quit1714@reddit | LocalLLaMA | View on Reddit | 11 comments
What kind of CPU, RAM and Motherboard would you recommend for a setup with the 6000 Pro? I want to get the Maximum performance out of it. Which cooling system is good? Would be grateful for input and short discussion.
doradus_novae@reddit
If your goal is to do more than 1 card eventually do yourself a favor and get an Asus WRX90 Pro Sage.
I use a threadripper and an artic 60$ cooler i bought off amazon. works fine.
kryptkpr@reddit
Zen5 EPYC with 8 CCDs and 12x DDR5 dimms
You may need to give up a kidney as prices on this gear have ~tripled this year
I am rocking an older Zen2 (7532) with no regrets, even older EPYCs are so much better then any consumer cpu.
eloquentemu@reddit
This is bad advise. If you want to run large MoE models, Epyc can't be beat (for the price, even now), but in that case you get 90% of the performance with a 5090 (maybe even 4090) as you are with the 6000 PRO. The 6000 is basically a luxury for long, unquantized contexts at that point (and yes, I own and have tested this).
If you just want to run the 6000 PRO with some model that fits in 96GB of VRAM, an Epyc is a waste of money and you should just run whatever cheap system you have lying around. Well, maybe PCIe Gen3 minimum :)
kryptkpr@reddit
It's not about context, it's about being able to offload experts without performance taking a shit. When I upgraded from xeon v4 to Zen2 I gained 10-15% on my shit 3090s so if you decide to pick an ancient host for a pro 6000 it will bottleneck.
suicidaleggroll@reddit
It depends entirely on what size models you want to run. If everything will be <80 GB or so, then it doesn’t matter as the CPU will be idle anyway. If you want to run larger models, then it does matter as you’ll have to offload more and more of the model into CPU+RAM.
Conscious_Cut_6144@reddit
For a single Pro 6000 is basically doesn't matter within reason.
Get a regular current gen desktop with a large NVME SSD (and a very good PSU).
If you want to support multiple Pro 6000's it can start to matter a little depending on inference stack.
joninco@reddit
There's really one main question -- will you run models larger than the 96GB vram or not. If so, you need system ram to handle that. So figure out how much system ram you'll need to run your models. If more then 256GB ram, you need to go server grade epyc/xeon. Ram is stupid expensive right now.
If not, any consumer hardware will do with pcie 5.0x16, with a decent size nvme ssd. I bet you could find some deals on cpu/mobo/32gb ram right now. No special cooling required, just decent case airflow -- the 6000 pro ws has pretty good cooling even running at the full 600watts.
Western-Source710@reddit
Ultra 8 285K or Ryzen 9 9950X3D
SillyLilBear@reddit
You don't need much, I run dual RTX 6000 Pro on AM4 with very little performance difference from latest cpu/mb.
vhthc@reddit
Epyc 9565+ processor to be able to have as much ram possible, so you can offload huge MoE models to ram. Chassis and mainboard to extend on gpus in the future. Good chassis + extra fans to get rid of the heat. Cannot recommend a specific mainboard and case sadly, we went with a rack solution with supermicro which is too expensive imho
fizzy1242@reddit
More ram is always nice for loading model faster and if you ever need to offload to ram for MoE models. You don't need the best cpu and motherboard in the market, but don't get the cheapest ones either.
I would get a motherboard with 1 or 2 additional pcie slots if you ever want to expand and add more gpus.