Help with Proxmox + Debian + Docker /w Nvidia 5060TI
Posted by EarEquivalent3929@reddit | LocalLLaMA | View on Reddit | 14 comments
Hi! Im at my Witts end here. I've been trying for the past few days with varying levels of success and failure. I have proxmox running with a Debian VM running docker containers. I'm trying to use a 5060ti in passthrough mode to the Debian VM
I have the cpu set to host and passed through the 5060TI using PCI.
I'm super confused, I've tried following multiple guides. But get various errors. The farthest I've gotten is running the Nvidia official installer for 575. However nvidia-smi in the Debian VM says "no devices found". But I do have a device in /dev/nvidia0.
My questions are:
What (if any) drivers do I need to install in the proxmox host?
What drivers do I need in the guest VM (Debian)?
Anything special I need to do to get it to work in docker containers (ollama)?
Thanks so much!
Fredrik444@reddit
Hi,
VM proxmox config: https://imgur.com/1K1rvga
I used ubuntu 24.04 lts, this is how I installed the nvidia drivers on the vm and made it docker gpu compatible:
sudo apt update && sudo apt full-upgrade -y
sudo reboot
lspci | grep -i nvidia <- checks if you see the GPU inside the VM, if not double check that you have configured the passthrough correctly
sudo apt install nvidia-driver-570 -y
sudo reboot
nvidia-smi (checks if nvidia driver is running)
Make VM docker compatible:
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/nvidia.gpg
distribution=$(source /etc/os-release; echo ${ID}${VERSION_ID}) && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu24.04 nvidia-smi <- if this gives you OK, everything is good.
EarEquivalent3929@reddit (OP)
Thanks for the help, it seems to be working now! I did everything you said except I couldn't install nvidia-driver-570, it told me to do nvidia-open-driver-570, so I did and that seems to have fixed everything. Not sure what the difference was. It seems to be working with my ollama docker container too. I haven't actually checked token count yet to make sure it's performing as good as it should be, but it seems alot faster than my previous 6750xt so it's probably a good sign
Thanks for your help!
Willing_Landscape_61@reddit
I got this working pretty easily if you are ok with "pass through" configuration. Driver blacklisted on host.
EarEquivalent3929@reddit (OP)
Sweet! What driver are you using?
Willing_Landscape_61@reddit
https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html#id39
Compute only for Debian
Secure_Reflection409@reddit
Maybe run LXC straight outta Proxmox?
EarEquivalent3929@reddit (OP)
I would but unfortunately I can't. I also use the gpu to drive motion detection in frigate in another docker container
zipperlein@reddit
U can run docker inside an LXC container.
JayTheProdigy16@reddit
I did this myself passing through dual 3090's and it was one hell of a cunt of a process lmao. Gotta blacklist the drivers from the host, run Q35 with seaBIOS and ensure you're passing the raw PCIE device set as primary with all functions and then begins the fun part. in my case since i was passing in 2 GPU's, i had to specify a romfile for the VM to load to attach the GPU properly, otherwise only one or the other would would and not both GPUs which probably isnt a problem for you. The Nvidia drivers were also a cunt for me and would never work when i manually installed them so you can either install ollama which will auto-install the drivers for you or install the CUDA toolkit which IIRC also installs the normal graphics drivers. You dont need any nvidia drivers on the host machine. You also mentioned running dockerized ollama which is technically doable but for the sake of my sanity i left it at 1 layer of virtualization, you should be able to install docker + Nvidia container toolkit and be good to go but there really isnt any inherent benefit over already being virtualized with Proxmox
zipperlein@reddit
What you can do also is use LXC-containers with the Nvidia container toolkit. Then you don't have to do any passthrough and you can share the GPU between multiple containers. And you can run docker inside a LXC container, if u want to.
EarEquivalent3929@reddit (OP)
Ah I'm not running seabios. I wonder if I can just install ollama on the guest for the drivers and then disable the service and attempt to use my docker container
MixedPixels@reddit
Do you have an integrated gpu on the motherboard? Are you plugged into it? then blacklist the driver so it isnt used when the os boots.
If not (you don't have igpu) you may need to 'switch the driver off, then login using the web based method' This is so you are not loading the driver when the os boots up, and it is available for the vm to use the driver when IT boots up.
https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_host_device_passthrough
EarEquivalent3929@reddit (OP)
Ah I'll try this. I am plugged into the motherboard but I have the monitor turned off. I didn't think it would be an issue since I had already tested seperate windows vm working perfectly (detected the GPU and ran VR games) with GPU passthrough as a test (I've since deleted that just to make sure it wasn't causing conflicts).
zipperlein@reddit
What can help is to make sure the audio part and other functions of the GPU within in the same IOMMU group are passed through aswell. Should be something like XX.XX.1, XX.XX.2 and so on.
Check your bios if you can make the iGPU the primary display device (if you have one).
As stated in the wiki: If u use lscpi -nnk it should list vfio-pci as used driver.
Make sure options related to virtualization support in your BIOS are set to enabled. (For example: VT-d or AMD-V, Above 4G Decoding (easy to forget), eventually Intel VT-x)