Where to Begin?

Posted by susne@reddit | LocalLLaMA | View on Reddit | 2 comments

Hey there I'm gonna be starting out on a 4080 mobile (12gb vram, 32gb ram, 14900hx) while I finish my 7900xtx desktop build and would like to know a few things. Which version of LLaMA should I start out with on the 4080 mobile? I think it can handle 13bP, I want to just get a feel of the possibilities and setup a TTS that can view my screen and chat for starters. What distro(s) of Linux are ideal and why? I will be using Windows 11 Home and want a Linux distro to contrast and compare experiences on both.

2 Comments

[-]

maddogawl@reddit

I think you’d probably want more like the 7b params on the 4080 mobile. Or you can pick a lower quantization, for Q4_k_m a 7b model with decent context should run well. It’s all a trade off. I out this together on how to pick and understand what sizes you can run. https://youtu.be/M65tp0EvLNo

Glittering_Mouse_883@reddit

I would suggest Ubuntu for your Linux distro. I see you are building a desktop with 7900xtx so you will need to install rocm to get that working well. Basically all the beginner tutorials for rocm are for Ubuntu, so you might as well get used to using it now so you'll be nice and comfortable with it by the time your desktop is finished. Once you have Ubuntu just get ollama and you'll be able to run a lot of stuff on your laptop. I'm pretty sure you can install ollama on windows too, but I think you're better off on Linux in general if you want to get into this.

Reply to Post

2 Comments