Secondary PC options
Posted by UniqueIdentifier00@reddit | LocalLLaMA | View on Reddit | 8 comments
Hey everyone, I’ve been lurking here for while.
I’ve really been enjoying messing around with my 6gb card on my laptop using Gwen 3.5 4B, ollama, and Open WebUI.
One of my friends is gifting me his old PC. It currently has a 3060 8gb in there, and only 8gb of RAM. I’d like to throw Ubuntu on that and use it as a server setup so I can locally access an LLM over my home network. Looks like I can easily do that with my dockerized OpenWebUi setup I’ve been using on my laptop.
My main question is, given my extreme lack of experience in regard to LLM’s, how do I best go about upgrading this PC? The goal is to be able to run 27B - 30B models. I could buy a 3090, but that alone won’t be enough. I could also use the 3060 alongside that to get enough usable VRAM, but I understand there are complications with tying multiple GPU’s together, something related to offloading that I don’t fully understand.
My other consideration is that I could buy 3-4 3060 12gb cards for the price of a single 3090. I don’t know what all it would take power and rack wise to be able to set that up, let alone how to properly use PCI lanes for that.
Next issue is RAM. How much actual RAM do I need to have to be able to use some of there bigger models? I was under the impression that VRAM is what matters, not RAM, to a certain extent.
Thanks for reading and I hope somebody who’s traveled this path can lend a hand. I’m just trying to find the most cost effective way to be able to use some of there larger models. Take care.
rainbyte@reddit
If I was you I would keep the 3060 for small models (eg. Qwen3.5-9B) and then buy the biggest GPU possible (eg. 3090) according to the mobo and PSU specs.
Keep in mind 1x24gb GPU is better than 2x12gb, now there are even "affordable" 32gb GPUs.
kevin_1994@reddit
My dad bought a cheap ddr4 gaming PC with 32GB ddr4 and Ryazan 2700x for $300 CAD. Then i found him 2x3060 for $500 CAD. So $800 total.
It runs pretty well. Runs qwen 27b iq4 at 20 tok/s and qwen 3.6 35b iq4 at around 45 tok/s (with a couple layers on CPU).
Not bad for 800 CAD all in. That's like 500 USD
UniqueIdentifier00@reddit (OP)
That’s the thing, I could buy 2 x 3060’s for half the price as a 3090 and have the same amount of VRAM. More trouble though for sure. I’ll think it over. Thanks for sharing your experience!
kevin_1994@reddit
Multi GPU support is super easy in llama.cpp. the main worry with 3060s is that they're about 1/3 to 1/2 as powerful as a 3090, and then there's also performance overhead from multi-gpu. But they also are usually 1.5-2 slot cards and don't take a lot of power, so easy to add like 3-4 in a consumer PC case. Pros and cons lol
UniqueIdentifier00@reddit (OP)
Gotcha gotcha thanks. I guess I’m leaning towards a 3090 in my future
youcloudsofdoom@reddit
Personally I'd just buy the 3090 and run q36 27B on it, as per this: https://github.com/noonghunna/club-3090
You can really get tons done on just one 3090 these days, with minimal set up complexity.
UniqueIdentifier00@reddit (OP)
Okay, for some reason I had in my head that a single 3090 alone wouldn’t be able to run that. That certainly makes things a lot more simple, if less fun than a pile of four 3060s!
Thanks for the response and link 🤙🏼
youcloudsofdoom@reddit
Yeah, honestly for the cost 2x 3090 is a luxury, not a necessity - but one certainly is, in my experience (disclaimer - I do have 2x 3090s!)