Anybody have a stable Nvidia driver version running with 2x3090 on Ubuntu?

Posted by PassengerPigeon343@reddit | LocalLLaMA | View on Reddit | 17 comments

I’m having some major frustrations getting a new system set up. OS is Ubuntu 24.04.2 LTS; motherboard is MSI B860-P plus WiFi; and I have installed 2x3090s. I am using the integrated graphics for my monitor since I want the GPUs dedicated for compute. Both 3090s are detected and I can see them in my system info but they aren’t being utilized. When I select the Nvidia 550 driver (from the Additional Drivers GUI) that says “(proprietary, tested)” it installs no problem but then my system has a meltdown over two reboot cycles after. One reboot I lose my Ethernet and WiFi completely and integrated graphics drops down to 1080p. The GPUs work great in OLLAMA on that first reboot but everything else is falling apart. A second reboot and the system will no longer boot into the OS. I’ve reverted using Timeshift and repeated again with the same result. The second time through I tried to switch back to the x server driver but it still wouldn’t boot the third time. I’m hesitant to keep clicking buttons without an idea of what is happening or at least a better way of saving it than doing a full Timeshift recovery. Should I try a 535 version? Maybe a server one? Should I install using the command line? Any help would be greatly appreciated. The dual 3090 setup seems to be quite common here so I’m hoping someone has a stable setup running that I can learn from.

Reply to Post

17 Comments

[-]

munkiemagik@reddit

Mate I cant believe I am here struggling through the same. I've had no issues wiht 4090 or 5090 in an LXC runnign ik\_llama or llama.cpp But I now have 2x3090 to put in the machine and This time I want it in an Ubuntu desktop and I am literally on teh verge of tears trying to find a way to get thsi sorted. The daul 3090s work and are up and running with driver 580 BUT no matter what I do I cant adjust the fans on the two cards and the noise is driving me insane. Ive tried Ubuntu 24 adn 22, dirver 580,570,550,535 through ubuntu addioional drivers and apt. The one thing I havent tried is the same way I had to do it in proxmox with a 5090, I download and `run` the installation files manually. Oh well off I go again for anohter couple hours of chasing my own tail.

[-]

PassengerPigeon343@reddit (OP)

**UPDATE:** I got it working on Ubuntu 24.04.2. I took a lot of the different pieces of advice here and ultimately found a solution that worked. **TL;DR: I installed the 570 headless driver from the NVIDIA network repository. Also installed CUDA 12.8 and 570 utilities. Everything works now.** Although I can't say with certainty, I think what was happening was that I was installing a full display driver with everything included, which changed a lot of things on the system. I had manually turned the integrated display back on in the BIOS and was plugged into that, and I think that the integrated display drivers were written over or something and causing everything to go crazy. Just a guess though. Here is what I did to get everything working: First I installed CUDA 12.8 using the method [here](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=24.04&target_type=deb_local). While that installed CUDA successfully, it still didn't get me all the way there. I was expecting it to because of information I found in NVIDIA's documentation that indicated that starting in 12.8 it automatically installed the compute-only portion of the driver. It installed a toolkit for 570.86.10, but didn't seem to install the driver too and I was still not utilizing the GPUs and still didn't have nvidia-smi. I went on to follow the distro-specific instructions for network repository installation found [here](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html). I had to tweak the keyring URL to get that part working and I **stopped** before getting to the *sudo apt install nvidia-open* command: Instead I ran a search using: `apt-cache search nvidia-headless` and found a good option that matched what I had installed with CUDA (a 570 headless open driver). I confirmed it was an NVIDIA repository package with the first line and then installed with the second: `apt-cache policy nvidia-headless-570-open` `sudo apt install nvidia-headless-570-open` Then to close things out, I installed the utilities since they were still not installed which gave me nvidia-smi: `sudo apt install nvidia-utils-570` Everything is running perfectly and I'm fully utilizing the GPUs now. Thank you everyone for the help.

[-]

AnimeFanBois@reddit

Bro tysm I thought I need this so much! I was trying to get it working with 5090 but only driver 570 supported the 5090, and I cannot do the usual 550 + cuda 12.4

[-]

PassengerPigeon343@reddit (OP)

Glad to hear it! After the struggle I had with it, I was hoping this would help at least one person who came across this post in the future.

[-]

Guna1260@reddit

I am using 560 and has not found any issues

[-]

Such_Advantage_6949@reddit

I am using 535. But i think u should try to lower your ubunto to version 22. I had some driver issue before with 24. I recall there is some page on nvidia website on their compatibility for cuda and i remember it doesnt support 24 yet

[-]

mvLynn@reddit

Second this. I originally started on 24 and had a number of issues on a 3090s setup. Don't remember if any were driver based, but all around it was just troublesome. Switched to 22 and it's been smooth sailing, no issues with 535 or 550.

[-]

PassengerPigeon343@reddit (OP)

Thank you both for this advice, I may make a test environment on an external drive to see if I can test the theory that they will work fine on 22. If it works I may have my answer. I’m hesitant to do it only because I have already set up a lot of things that I’m hoping to not have to repeat on a fresh install but I may be forced to do that as a last ditch effort if I can’t get this working.

[-]

mvLynn@reddit

That was my hesitance as well when I realized 24 wasn't working out too well. Even though it had only been a week, I had already configured a number of other things, and as I'm not exactly a Linux power user, it felt like a slog to have to figure it out again. But in the end I was glad I bit the bullet as 22 is so much better. Just copy your terminal history from 24 and that server has a baseline to speed things up when you setup 22. Then make a guide as you go long with your new 22 setup and note down every command, every package, and every config you do. Makes it very simple if you have to start over from scratch. I think I can now get a new 22 install setup to run LLMs in like 30 commands.

[-]

PassengerPigeon343@reddit (OP)

You’re probably right about that, hopefully the second time through I can set things up a lot faster. I am also not a Linux power user (quite the opposite in fact) so everything I’ve managed to get working so far has been a struggle. Now that I have it working though, I should be able to go back through my steps and create my own documentation to follow. I still dread it but it may be my best option.

[-]

justintime777777@reddit

I run 550 on 24.04 on several 3090 builds without issue, But mine are all headless servers. Maybe try blacklisting nouveau

[-]

PassengerPigeon343@reddit (OP)

Thanks for this datapoint. I’m trying to understand where I am going wrong… I have a few questions if you don’t mind answering: - Are you using the regular driver version or a server version? - Do you install from the command line or using the “additional drivers” GUI in Ubuntu? - is this done with any other things alongside the driver itself like CUDA? I can try blacklisting nouveau if you think that could help. It seems like it is booting and functioning fine with the nouveau driver but once I pick the 550 instead it starts to self-destruct.

[-]

justintime777777@reddit

I find I have better luck just doing apt install Nvidia-driver-550 And yes, depending on what project I’m working with either apt install nvidia-cuda-toolkit (12.1?), Or some goofy workaround to get 12.4 Haven’t used 12.8 yet, but that one should install correctly on 24.04 from instructions on Nvidias site

[-]

suprjami@reddit

Whatever you run, make sure packages `nvidia-driver nvidia-cuda-dev nvidia-cuda-toolkit` all come from the same place. Don't mix distro packages with nVidia repo packages. I use Debian and had to start from scratch a few times until I got it right. `sudo apt purge dkms *nvidia*` and a reboot should get you back to a clean slate running nouveau so you can try again.

[-]

PassengerPigeon343@reddit (OP)

Where would you recommend I get these all from to ensure they match up? I am new to all of this so all I have done so far is to select the driver in the Additional Drivers GUI in Ubuntu but it’s possible I need to take a different approach. Or perhaps I need to install CUDA and the Toolkit first before the driver. And lastly after my system goes nuts and won’t boot, if I can boot into a command line rescue mode do you think that command will potentially save me without having to do a Timeshift reset? That would be a huge help if so.

[-]

Glittering_Mouse_883@reddit

I know this might not be helpful, but I'm running MX Linux which is Debian based like Ubuntu. They have a really easy Nvidia driver installer app that comes with the distro. Never had any issues and everything just works out of the box. I used to be on Ubuntu for a while but just got tired of all the random bugs - don't regret switching. So yeah if your Ubuntu install is already screwed up anyway you might consider just switch distro to something that will install the latest Nvidia driver with no issues.

[-]

PassengerPigeon343@reddit (OP)

Thank you. It is helpful information for sure. I am considering switching either versions (like other comments suggested) or to a different distribution all together. My only resistance is all the stuff I already managed to get working and I don’t want to have to start from scratch. Though it may be less headache than this is turning out to be.