Promox or Native Ubuntu
Posted by pipaman@reddit | LocalLLaMA | View on Reddit | 14 comments
I've just bought a new machine with 2 NVIDIA 3090 to run Llama.
I want to get advise if it is worth to use Promox or I will get most of the hardware just installing an Ubuntu.
-my_dude@reddit
You haven't told us anything you plan on running beisdes llama so going off that I'd say Ubuntu
pipaman@reddit (OP)
I want to use Llama, but in the future I may want to use the computer for something else, maybe gaming.
-my_dude@reddit
I don't know how you plan to game off Proxmox, it's a hypervisor. I recommend Ubuntu still.
pipaman@reddit (OP)
My plan is to install a Windows VM too
-my_dude@reddit
Proxmox isn't a bad option if you want to have more than 1 VM
caetydid@reddit
proxmox will be adding a minor performance overhead if you use pci passthrough and virtio. but you will be able to run multiple VMs and setups cleanly separated. or you install the nvidia drivers and cuda on the proxmox host and use lxc containers to run dockerized containers. then all containers can access your gpus. VMs accessing your gpu will exclusively lock them.
btw, ive got also two rtx 3090 and went for proxmox and VMs.
ThunderousHazard@reddit
This is the way.
Proxmox + LXC containers, no performance hit that I can measure on GPUs due to the direct binding in the container.
I am currently using this with two 3060s 12GB, and the neat part is that you can share the GPUs with as many containers as you like, meaning I am using the same GPUs with both the machine learning containers and the jellyfin container (for video transcoding).
Performance impact is absolutely not a concern, as lxcs are lightweight as hell (you don't virtualize the whole stack, using directly use the host kernel).
pipaman@reddit (OP)
Looks like this is a winner. I will use this setup.
ThunderousHazard@reddit
Well, most motherboards do support IOMMU, the problem is that the often the motherboard makers don't implement them very well.
I suggest you give a read to this section: https://pve.proxmox.com/wiki/PCI(e)_Passthrough
Also, note that IOMMU is needed only for VMs passthrough, if you use LXC containers there should be no need for it.
pipaman@reddit (OP)
I will use LXC containers.
ThePixelHunter@reddit
Proxmox LXC containers are the way. If you can get over the hurdle of setting up drivers twice - on the host, and in each container (hint, make a template!) - then you benefit from a clean separation for different tasks. It's worth the time to setup.
MoodyPurples@reddit
I went with bare metal ubuntu for my dual 3090 server and now I’m wishing I had went with Proxmox, but not enough to reinstall yet. A container I wanted to run needed a higher version of Cuda and if I had proxmox I could make a new VM and test the rest of my setup on that version before committing to it.
Secure_Reflection409@reddit
Native.
Blindax@reddit
Not sure about it but passing through the two GPU may let your proxmox server headless unless you have an iGPU.
Unless you are already comfortable with proxmox and GPU passthrough I would definitely go with Ubuntu or pop os if you want to avoid the headache.