[Help] Running DPDK on proxmox, inside VMs using SR-IOV to forward VFs to VMs and bind the VF to dpdk inside the VM.

Posted by Ginnungagap_Void@reddit | sysadmin | View on Reddit | 4 comments

Hello!

We've been trying for a week and a half to make this project happen

Use SRIOV to split the Intel X520 DA2 into 4 virtual functions per port and forward each VF to a virtual machine.

In the VM bind this interface to dpdk and use it in OVS

As we didn't manage to get it working in the VMs we bound the VFs on the host and they worked

However when inside the VMs, no interface works, dpdk-testpmd says the interface has 0 RX or TX queues.

We updated the ixgvevf and ixbge drivers on both the host and VM to the latest from Intel, and, we use the same 6.8 kernel on both host and VM.

The server we use now is a DL380 Gen9, that has SRIOV support in firmware.

I believe we got the virtual machine's settings wrong.

We set it up as a Q35 machine with Intel virtual IOMMU enabled.

Boss is breathing down our neck and we're getting desperate now.

Can anyone give us a hint? Why would the VF inside the VM be seen while no RX and TX queue be available?

We had the same problem on the host and installing the latest ixgbe and ixgvevf driver solved the issue.

We compiled both DPDK and OVS from source.

[-]

BakedWatchingToons@reddit

Well above my head, however I wonder about how the vm knows the VF is there if it's bound to the host only, if that makes sense?

My thinking is just because there is a physical "thing" on the host, that doesn't at all mean that same "thing" (sorry for the shitty and non technical words) exists on the vm.

Let's say it's a usb device. That's easy. When attaching a usb to a vm, there's a setting to specifically pass it through. Similar with vhds etc, pretty sure.

Is there a specific setting you've muddled with to tell the vm there's a usb thing you're passing through?

Hope my effort wasn't condescending or shitty! Again, freely admit this is a touch beyond me haha

Ginnungagap_Void@reddit (OP)

The VF of the NIC is passed through to the VM :)

Nic is connected to host, plit into 4 by SRIOV and one virtual functions forwarded to the VM.

IdiosyncraticBond@reddit

From a quick search, coukd https://forum.proxmox.com/threads/support-issues-with-sr-iov-on-proxmox-8-2-2.149365/ help?

Yes, that's how we found out we need the updated ixbge and ixgbevf drivers.

And sriov worked on the host from that point on

Right now it doesn't work on the VMs