Best small local llm for laptops

Posted by The_GSingh@reddit | LocalLLaMA | View on Reddit | 13 comments

I was wondering if anyone knows the best small llm I can run locally on my laptop, cpu only. I’ve tried out different sizes and qwen 2.5 32b was the largest that would fit on my laptop (32gb ram, i7 10th gen cpu) but it ran at about 1 tok/sec which is unusable. Gemma 2 9b at q4 runs at 3tok/sec which is slightly better but still unusable.

Reply to Post

13 Comments

[-]

HansaCA@reddit

Deepseek v2 lite runs surprisingly quick on my no-gpu laptop. Faster than other same size models. Perhaps because of MoE?

[-]

The_GSingh@reddit (OP)

Damn tried it out and way better than what I was using before Gemma 9b. Gemma ran at 3tok/sec, while deepseek coder v2 lite ran at 8tok/sec making it useable

[-]

jamaalwakamaal@reddit

I used it after this comment and I am gobsmacked that I can use deepseek codes v2 on cpu

[-]

The_GSingh@reddit (OP)

Yea it’s a MoE which means all 16b params are loaded into memory but actually 2.4b are used in inference. For non-MoE’s it’s just all the params getting used so for something like qwen 2.5 14b all 14b are loaded into memory and all are used.

[-]

jupiterbjy@reddit

I had exact same thinking before as my laptop ships with crap called 1360P w/ 32GB ram. Ended up using Qwen 2.5 3B coder + llama 3.2 3B + OLMoE for offline inferencing in flight as none of single model was best fit for all usecase. For cpu inferencing while utilizing ram you have, MoE models are real nice fit - but well, problematic part is that it's rare. OLMoE is the only sensible looking thing to me as other models are either too large, only MoE of two model, or too small. OLMoE runs quite fast on cpu thanks to it being 1B Active param w/ 7B total size but feels not particually trained long enough - try this model as last ditch effort if all other small models disatisfy you.

[-]

The_GSingh@reddit (OP)

Btw are you referencing OlMoE 1 or 2. The first one from what I can tell couldn’t even compete with last gen open llms

[-]

jupiterbjy@reddit

There's no 2 in olmoe yet, maybe you confused with OLMo which isn't MoE - still OLMoE isn't good for it's size which is why I think it's last ditch effort. MoE models are so underrated and less researched.. sigh

[-]

Ok_Warning2146@reddit

MoE models are bad for Nvidia gpus due to high VRAM usage. But they are good for PC and Mac when you have a lot of RAM.

[-]

jupiterbjy@reddit

Yeah exactly as OP described, have much leftover ram, capped by cpu.

[-]

The_GSingh@reddit (OP)

Yea it varies wildly. Most models I can’t even get above 2tok/s. Tried qwen 2.5 0.5b and managed to hit above 30tok/sec which was insane but naturally the model sucked. I’m just surprised that between the 32b and 3b models there’s a small difference and that even the 1b models run at the same speed as 7b ones for me in 4bit.

[-]

supportend@reddit

Depends on the tasks, some models are better at summarize, others in xyz. You could test gemma-2-2b and Llama-3.2-3B-Instruct.

[-]

PassengerPigeon343@reddit

These two have been the best tiny models so far in my experience for general chat. Gemma 2 9b has been my absolute favorite and best all around for my use, but I would not enjoy 3 tok/sec. I find the Gemma 2 2b better at creative tasks and LLaMA 3.2 3b seems to be better at answering questions correctly. I would use both of these and see what works better in your use cases.

[-]

suprjami@reddit

Granite 3.1 is currently very high on the [GPU Poor LLM Arena](https://huggingface.co/spaces/k-mktr/gpu-poor-llm-arena), give it a try: https://huggingface.co/bartowski/granite-3.1-2b-instruct-GGUF