The issue with Gen 4 isn't just lanes. Until very recently, you could get epyc boards for less than the cost of that PLX card. Even now, you can find the occasional deal if you're a bit savvy.

The thing with Gen 4 is that it's a lot more sensitive to noise. So, your cables also have to be Gen 4 (more expensive) and your riser needs to be able to handle Gen 4. And all this for what? 3090s don't have p2p anyway. And we haven't even talked about how so many consumer boards crap themselves when more than two GPUs are connected, even ignoring lane count.

If you go with an Epyc, you get 128 lanes, many boards expose 90+ of those on PCIe slots. At X8 per GPU, that's 12 GPUs without sweat, while having ~16GB/s dedicated bandwidth from the CPU to each GPU. You'll absolutely need that without p2p because all communication has to go through the CPU (and the code running on CPU). With a PLX, bandwidth to the CPU quickly becomes a bottleneck.

You'll actually see worse performance using that PLX card, because all your GPUs will have to use that single x16 link to the CPU, whereas now you have 20 lanes. And you're lucky your motherboard didn't crap itself out already with three GPUs.

I have four 3090s in a Epyc system, and I can see 25GB/s per GPU running models across all four using ik_llama.cpp. That's 100GB/s (byte, not bit) flowing in and out all four during inference.