If you have 6GB or less VRAM and 16GB RAM, don’t go over 8B parameter models. Anything larger (especially models over 6GB in download size) will run very slow and feel sluggish during inference, And can damage your device overtime.

🔍 After lots of testing, I found the sweet spot for my setup is:

8B parameter models

Quantized to Q8_0, or sometimes FP16

Fast responses and stable performance, even on laptops

📌 My specs:

GTX 1660 Ti (mobile)

Intel i7, 6 cores / 12 threads

16GB RAM

Anything above 6GB in size for the model tends to slow things down significantly.

🧠 Quick explanation of quantization:
Think of it like compressing a photo. A high-res photo (like a 4000x4000 image) is like a huge model (24B, 33B, etc.). To run it on smaller devices, it needs to be compressed — that’s what quantization does. The more you compress (Q1, Q2...), the more quality you lose. Higher Q numbers like Q8 or FP16 offer better quality and responses but require more resources.

🔸 Rule of thumb:
Smaller models (like 8B) + higher float precision (Q8 or FP16) = best performance and coherence on low-end hardware.

If you really want to run larger models on small setups, you’ll need to use heavily quantized versions. They can give good results, but often they perform similarly to smaller models running at higher precision — and you miss out on the large model’s full capabilities anyway.

🧠 Extra Tip:
On the Ollama website, click “View all models” (top right corner) to see all available versions, including ones optimized for low-end devices.

💡 You do the math — based on my setup and results, you can estimate what models will run best on your machine too. Use this as a baseline to avoid wasting time with oversized models that choke your system.