Thinking to buy server chassis pcie 5.0 and 1x to 4x 3090

Posted by kidfromtheast@reddit | LocalLLaMA | View on Reddit | 13 comments

Should I buy a server chassis with pcie 5.0 and buy 1x 3090 to 4x 3090?

Hoping to find modded 3090 with pcie 5.0. not sure whether it exists. hmm, there is 4090 48gb mod, so why not a 3090 pcie 5.0 should not exist

last time i spent almost $100 on a single benchmark. I am starting to increase the number of benchmark. Considering 1x costs 3090, it might be a saving to own a GPU, not sure. Also, considering graduation from Master to the start of PhD will take 6 months of holiday, I want to fill the holiday with research, so, owning GPUs would be lift mind block (I live in Asia, renting GPUs is expensive, to the point I would not let myself sleep over it, and create a script to auto shutdown once the experiments are finished)

i don't have a workstation now. Only a laptop and I use GPU remotely either on Runpod or lab’s GPUs.

[-]

Pixer---@reddit

If you do. Threadripper wrx80 is a nogo, I tested that myself. I would not go with any threadripper tbh. You’ll want the p2p workaround driver for cuda, which needs a epyc cpu 7002/7003/9004/9005. For multi gpu the lower latency of p2p and the almost doubling of card to card copy is great. I would suggest the romed8-2t combo. You’ll need p2p to properly utilize your GPUs. Using tensor parallelism you have 120+ all reduces between GPUs. Without it takes around 10-20us but with p2p it’s like 0.5-2us. This is a bottleneck for multi GPU setups.

If you want to train AI models P2P is a must for multi gpu. And nvlink does gain in speed, but the latency is the most important one for training, which native p2p reduces for small copies, until you go into bandwidth limitations.

[-]

OnkelBB@reddit

I found your comment some time ago and it made me think of a wrx80 build limits.
However I might find a solution for TR builds in a form of pcie switch: https://github.com/local-inference-lab/rtx6kpro/blob/master/hardware/topology.md

[-]

DeathScythe676@reddit

if you're looking at 4x 3090's for LLM, you're already looking to save money so i wouldn't bother with a DDR5 server motherboard. I have 4x 3090's on a supermicro H11SSL-i DDR4 motherboard and it's more than fine. once models are loaded it barely touches the CPU for anything.

Be prepared for a lot of tinkering,

and you should take a look at your energy/electric rates,

and physical location planning. Where are you going to put it?

my 4x 3090 rig is about 1200w constant load when grinding away. And long projects it can be grinding for hours at a time.

You do not want this in your bedroom.

[-]

MelodicRecognition7@reddit

PCIe5 means that your server is designed for DDR5 RAM and it will cost multiple times more than the 3090s. Consider getting a newer generation cards if you plan to spend that much money.

[-]

Apart_Ebb_9867@reddit

PCIe5 means that your server is designed for DDR5 RAM

PCI standards versions and RAM standard versions have nothing to do with each other.

[-]

ImportancePitiful795@reddit

There are no workstation or server boards with PCIe5 and DDR4.

[-]

Apart_Ebb_9867@reddit

define workstation. I don't want to make a mountain out of a molehill, my only comment was that PCIe5 does not imply DDR5. This would be true even if all motherboards on this planet that have PCIe5 had DDR5. The fact they don't have doesn't mean they couldn't have that mix, because the two technologies are totally unrelated and the post I was replying to stated because of X, then Y.

Specifically, even though uncommon, motherboards with PCIe5 and DDR4 do exist, for instance this.

So, I'm out of this discussion. People can believe Pcie5 implies DDR5 if so they chose.

[-]

ImportancePitiful795@reddit

Ah yeah you found the ONLY DESKTOP motherboard having DDR4 and PCIE5.

Workstation, means WORKSTATION. Those motherboard taking WORKSTATIONS CPUs with 40+ PCIe lanes, having 4/6/8 channel memory and can connect 4 RTX3090 to them without requiring more than just put the card on the board.

[-]

Solary_Kryptic@reddit

Think they mean there aren't any server boards with PCiE 5 and DDR4

[-]

ImportancePitiful795@reddit

a) There is different thing to add VRAM and mod the BIOS than change the PCIE the CHIP ITSELF has.

b) Get an X399 with 1920X+ combo workstation board (around €400-€600). Use standard DDR4 RAM.

Make sure the motherboard has 4-5 PCIE slots. Plug the 4 x 3090s directly.

That's the cheapest solution.

Alternative by a QYFS + MS73 + 64GB DDR5-RDIMM bundle (€2500) and put those GPUs on it.

The latter has the benefit to use Intel AMX to boost matrix computations (ktransformers work well with it).

If you go down the latter path, consider to stick to 2 RTX3090s and buy as much RAM as possible.

Unfortunately if you had asked last year, would have sent you down that path without second thought, since 500GB RAM was around €1800, and with couple RTX3090s or a single 4090 could run whole full size DeepSeek R1 at Q4 at respectable speeds. But with the current DDR5 RAM prices that's prohibited.

[-]