hey! i double checked, p2p i had was through nvlink on my 3090s. looks like you do need to unlock p2p at the driver level with older cards. blackwell 6000s support it.

i also was doing more testing, turns out, you dont even need a switch to get p2p. from my AI systamin:

Good questions. For 2× RTX PRO 6000 on EPYC 7003 Gen4, you probably do not need a PCIe switch at all.

Your board likely already gives you two direct Gen4 x16 slots from the CPU, and for a 2-GPU setup that is effectively the same as, or better than, adding a switch. The switch only becomes necessary if you want to scale to 4+ GPUs later.

Also, small correction to my own framing: I had mentally conflated NVLink with P2P. NVLink is just one interconnect for GPU-to-GPU communication. PCIe can also do P2P, as long as the hardware, topology, driver, and kernel settings allow it.

Here is the breakdown.

1. How PCIe switches work, and whether you need one

You are right that PCIe switches (PLX was one vendor; today it is mostly Microchip/Microsemi and Broadcom) create a private GPU-to-GPU fabric. When GPU 0 writes to GPU 1 through a switch, the data never touches the CPU. The switch routes it directly between downstream ports.

But the key point is this: direct-attach topologies do the same thing, just through the CPU root complex instead of switch silicon.

On a modern server CPU like EPYC 7003 with 128 Gen4 lanes, each GPU slot is a real x16 directly off the CPU, and P2P DMA between two slots on the same CPU works correctly. The CPU is not really "in the way" here. It behaves more like a passive router.

So at exactly 2 GPUs, the performance difference between:

• direct-attach
• PCIe switch

is basically zero, assuming both are configured correctly.

A switch becomes useful when you want 4+ GPUs in one system, because then you start running out of clean direct x16 slots and need fan-out.

Bottom line for your setup: skip the switch, use the board's native slots, and focus on kernel/driver config instead.

2. GPU requirements

You have this right.

RTX PRO 6000 supports hardware P2P natively. NVIDIA enables BAR1 peer-to-peer writes on workstation and datacenter cards.

Consumer cards like 4090 / 5090 have P2P disabled in the driver.

3090 is a weird middle case. It can sort of work through the UVM software path, but generally routes through host memory unless you use patches, so most people treat it as effectively disabled.

So for your setup:

• RTX PRO 6000: good out of the box
• consumer GeForce cards: not ideal
• 3090: partial / messy

3. What the kernel and modprobe settings actually do

CPU performance governor

Linux often defaults to ondemand or powersave.

During LLM inference, the CPU can go briefly idle between allreduce sync points. The governor sees that and downclocks. Then the next sync arrives and the CPU has to ramp back up again. That ramp latency can eat a noticeable chunk of decode throughput.

Setting the governor to performance keeps clocks pinned and avoids that behavior.

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

To make it persistent:

sudo tee /etc/tmpfiles.d/cpu-governor.conf > /dev/null <<'EOF'
w /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor - - - - performance
EOF

pci=noacs

ACS (Access Control Services) is a PCIe feature mainly meant for virtualization and isolation.

The problem is that ACS can force cross-device traffic back up through the CPU root complex in a way that hurts GPU-to-GPU P2P behavior.

pci=noacs disables that behavior and lets the GPUs DMA directly to each other.

This is useful on a trusted host where you care about performance more than strict isolation.

On Ubuntu/GRUB, that means adding it to the kernel cmdline and rebooting.

uvm_disable_hmm=1

HMM (Heterogeneous Memory Management) is a newer path in NVIDIA UVM that can migrate pages between system RAM and VRAM.

That is useful in some cases, but under sustained P2P DMA load it can interact badly with the driver and produce the classic failure mode where the system works for a while, then a GPU wedges into ERR! state and falls off the bus.

Disabling HMM forces the older, more stable path.

echo 'options nvidia_uvm uvm_disable_hmm=1' | sudo tee /etc/modprobe.d/uvm.conf
sudo update-initramfs -u

This is one of those bugs that is hard to diagnose if you do not already know about it.

4. The extra thing that was missed originally

For direct-attach topologies like a normal 2-GPU EPYC system, there is one more modprobe setting worth knowing about.

Without it, SGLang's custom allreduce kernel can silently fall back to a much slower path.

# /etc/modprobe.d/nvidia-p2p-override.conf
options nvidia NVreg_RegistryDwords="ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1"

What this does is force NVIDIA to use BAR1 P2P writes instead of host-memory staging.

Without it, --enable-pcie-oneshot-allreduce may auto-disable itself for small message sizes and fall back to NCCL.

With it, the custom allreduce path stays active and performs much better for the decode-sized allreduces that actually matter.

Important: only do this on direct-attach / NODE systems.

If your topology is PIX / PXB because you are using a real PCIe switch, do not apply it.

Check first:

nvidia-smi topo -m

Rule of thumb:

• NODE / PHB → apply ForceP2P override
• PIX / PXB → do not apply it

Verify after reboot:

cat /proc/driver/nvidia/params | grep RegistryDwords

5. Complete setup sequence

Assuming Ubuntu 22.04 and starting clean:

# 1. Check topology
nvidia-smi topo -m

# 2. Add kernel parameters
sudo nano /etc/default/grub
# Add:
# pci=noacs iommu=pt pcie_aspm=off mitigations=off
sudo update-grub

# 3. Disable UVM HMM
echo 'options nvidia_uvm uvm_disable_hmm=1' | sudo tee /etc/modprobe.d/uvm.conf

# 4. ForceP2P override (NODE/direct-attach only)
sudo tee /etc/modprobe.d/nvidia-p2p-override.conf > /dev/null <<'EOF'
options nvidia NVreg_RegistryDwords="ForceP2P=0x11;RMForceP2PType=1;RMPcieP2PType=2;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1"
EOF

# 5. Performance governor
sudo tee /etc/tmpfiles.d/cpu-governor.conf > /dev/null <<'EOF'
w /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor - - - - performance
EOF

# 6. Rebuild initramfs and reboot
sudo update-initramfs -u
sudo reboot

After reboot, verify:

cat /proc/cmdline
cat /proc/driver/nvidia/params | grep RegistryDwords
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
nvidia-smi topo -m

Also disable ASPM in BIOS for extra stability.

6. How to verify it is actually working

Build and run the CUDA sample:

git clone https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest
mkdir build && cd build && cmake .. && make
./p2pBandwidthLatencyTest

You want to see something roughly like this for Gen4 x16:

• Unidirectional P2P enabled: \~26-29 GB/s
• Bidirectional P2P enabled: \~46-48 GB/s
• P2P enabled latency: \~0.38 µs
• P2P disabled latency: \~14 µs

If enabled bandwidth is low and latency is still \~14 µs, something is wrong. Usually that means:

• ACS is still interfering
• ForceP2P did not apply
• uvm_disable_hmm=1 is not loaded

7. Final software-stack tip

Once the kernel side is correct, one of the biggest wins on 2× RTX PRO 6000 is running SGLang with the right container and kernels.

For Qwen3.5-122B NVFP4, people are seeing around 170-200 tok/s at single-user decode on tuned 2-GPU setups, which is substantially better than stock vLLM on the same workload.

So the practical conclusion is:

• For 2 GPUs on EPYC: direct-attach is already the right topology
• You do not need a PCIe switch
• The real work is kernel, driver, and software-stack tuning
• A switch matters later, when you scale past 2 GPUs