Multiple GPUs and supplying power to the PCIe slots
Posted by hainesk@reddit | LocalLLaMA | View on Reddit | 18 comments
For people using multiple GPUs in their system, like 3 or more, have you had to do anything special to make sure there is enough power supplied to the PCIe slots? Each one provides up to 75 watts per GPU, and it's my understanding that most consumer motherboards only provide around 200 watts to the PCIe slots, enough for 2 GPUs, but 3 or more and it gets dicey.
Pedalnomica@reddit
Personally, I'm using PCIe->SlimSAS->PCIe adapters, and the the final adapters are powered. https://c-payne.com/products/slimsas-pcie-gen4-device-adapter-x8-x16
(had bad luck with eBay knockoffs of this).
I've found this is much nicer to work with than the PCI-e Riser cables I used to use, but a bit more expensive.
FreegheistOfficial@reddit
same for me. tried cheap options and host of issues. using the c-payne hardware has been hassle free everything works
Pedalnomica@reddit
So, I will say I was able to save a decent amount of money using this 10GTek card on the motherboard side https://www.amazon.com/dp/B0C4P2PKJV (except in PCI slot two of my ROMED8-2T motherboard which needed a redriver and I used one from cpayne).
FreegheistOfficial@reddit
interesting. so that works connecting to the c-payne device adapters? like no errors / full bandwidth? What SAS cables are you using?
MelodicRecognition7@reddit
yea same for me, I've tried one ~50 EUR chinese copy and it did not work, so had to return it. But this 250 EUR option is way too expensive.
Pedalnomica@reddit
??? It is 50 EUR
MelodicRecognition7@reddit
it's just the card with 2x SlimSAS ports, to make a PCIe riser you also need a card with PCIe x16 port and a double SlimSAS cable, all three items together cost 250 EUR
hainesk@reddit (OP)
That looks great. I’m considering switching to all oculink via a x4x4x4x4 bifurcated pcie card. Something like this: https://www.amazon.com/LetLinkSo-PCIe-Oculink-SFF-8612-x16/dp/B0F291T2L4/ combined with these: https://www.amazon.com/gp/product/B0BZW1G87R/ which have worked well for me in the past. If I use those with the nvme connected gpu I think I can have 5 or maybe 6 gpus connected on a single consumer motherboard. But at that point 4 of the cards will be limited to 4.0 x4 speeds and the other two will share one 4.0 x4 chipset link to the cpu lol.
Pedalnomica@reddit
I haven't tried it, but some people say 4.0 x4 is fine. I just tried something with 3.0 x2 and it was not great, but I think that 1) might have just been the crappy computer in general as opposed to interconnect specific, and 2) that is 4x slower than you'll have.
__JockY__@reddit
I think some motherboards have an extra power connector on board to supplement the PCI rails. One example is the Gigabyte MZ33-AR1.
Having said that, I’d think that any GPU (with high-current dedicated PSU connector) must surely be poorly designed if it pulled significant power from the PCI bus? It would surely be a recipe for intermittent weird failures and crashes!
Feeling-Currency-360@reddit
Generally if you have a motherboard capable of housing multiple GPU's, you wouldn't use GPU's that run off PCIe power, you'd use GPU's that utilize external power via 6-pin/8pin etc
hainesk@reddit (OP)
GPUs with external power still pull from PCIe power.
Feeling-Currency-360@reddit
Your wrong I think, they might pull a small amount of power, but not the whole 75 watts the PCIe slot is rated for.
Lets say for argument sake that you connect 8 GPU's to an motherboard an power them all via external power.
Obviously the board will not provide each card with 75W via PCIe because it can't, but the cards have more than enough power available to them via their external connectors, so no problem.
It's only a problem if the cards don't have external power connectors and they all NEED to run off PCIe.
Honestly I'd love to have this sort of problem, generally more than 2 PCIe slots at 8x or more speed are a luxury, best you'll find on consumer boards are a 16x and an 8x slot but that doesn't leave much PCIe lanes for M.2 storage, consumer CPU's just don't have nearly enough PCIe lanes.
hainesk@reddit (OP)
GPUs are high power devices and will pull from both external PCIe power connectors as well as the PCIe slot.
Here is a video of someone testing a PCIe slot power measurement device as an example with a 2080ti: https://youtu.be/dSjQj4808uA?si=y2T5wzIZeLhkaVqj
It pulls around 60 watts of power while under load.
Lissanro@reddit
In the past, before I upgraded to a server motherboard, I had 4 3090 GPUs connected to a consumer motherboard instead, three of them directly via x16 risers, and one via x1 riser. It worked fine. In practice, GPUs do not draw that much power from PCI-E slot anyway. You just need to ensure they have enough power on their main power connectors and try to not go beyond 80% of rated PSU power, if necessary get additional PSU and sync them together using Add2PSU board.
The bigger concern on a consumer motherboard is usually lack of PCI-E lanes, which can slow down model loading, tensor parallel inference and ruin training/fine-tuning performance.
hainesk@reddit (OP)
This is what I have currently, 3 3090s (2 directly in pcie slots and 1 via riser) and 1 3090 via x4 nvme. I have a 1600 watt psu connected and I have some power issues that come up only when I try to use tensor parallel x4. Regular inference with TP x2 and PP x2 is ok, and regular inference with Ollama or llama.ccp is ok. Were you able to run TP x4 on your consumer motherboard?
Lissanro@reddit
No, tensor parallel did not work on the consumer motherboard, most likely because one of my cards was connected via x1 PCI-E 3.0. I was getting 15-20 tokens/s with Mistral Large 123 5bpw EXL2 quant in TabbyAPI, since only could use speculative decoding. After upgrading to EPYC motherboard with all four 3090 cards in x16 PCI-E 4.0 slots, I could enable tensor parallel inference and performance increased to 36-42 tokens/s for the same model.
That said, I never had any power issues, I could for example run image generation on all 4 GPUs, even if let them to use 390W power. It is likely your PSU is just not that great. Try perhaps leaving only one GPU on it, and connect other two to another PSU, using Add2PSU board to ensure common ground and that they turn on and off at the same time.
ducksaysquackquack@reddit
i have msi x670e tomahawk w/ 5090 in x16 slot 1 via riser, 4090 in x16 slot 2 via riser, and 3090ti in x16 slot 3 no riser. aida64 sensorpanel indicates the pcie slots themselves using no more than 25-50 watts each during inference.