CPU-only LLM performance - t/s with llama.cpp

Posted by pmttyji@reddit | LocalLLaMA | View on Reddit | 64 comments

How many of you do use CPU only inference time to time(at least rarely)? .... Really missing CPU-Only Performance threads here in this sub.

^(Possibly few of you waiting to grab one or few 96GB GPUs at cheap price later so using CPU only inference for now just with bulk RAM.)

I think bulk RAM(128GB-1TB) is more than enough to run small/medium models since it comes with more memory bandwidth.

My System Info:

Intel Core i7-14700HX 2.10 GHz | 32 GB RAM | DDR5-5600 | 65GB/s Bandwidth |

llama-bench Command: (Used Q8 for KVCache to get decent t/s with my 32GB RAM)

llama-bench -m modelname.gguf -fa 1 -ctk q8_0 -ctv q8_0

CPU-only performance stats (Model Name with Quant - t/s):

Qwen3-0.6B-Q8_0 - 86
gemma-3-1b-it-UD-Q8_K_XL - 42
LFM2-2.6B-Q8_0 - 24
LFM2-2.6B.i1-Q4_K_M - 30
SmolLM3-3B-UD-Q8_K_XL - 16
SmolLM3-3B-UD-Q4_K_XL - 27
Llama-3.2-3B-Instruct-UD-Q8_K_XL - 16
Llama-3.2-3B-Instruct-UD-Q4_K_XL - 25
Qwen3-4B-Instruct-2507-UD-Q8_K_XL - 13
Qwen3-4B-Instruct-2507-UD-Q4_K_XL - 20
gemma-3-4b-it-qat-UD-Q6_K_XL - 17
gemma-3-4b-it-UD-Q4_K_XL - 20
Phi-4-mini-instruct.Q8_0 - 16
Phi-4-mini-instruct-Q6_K - 18
granite-4.0-micro-UD-Q8_K_XL - 15
granite-4.0-micro-UD-Q4_K_XL - 24
MiniCPM4.1-8B.i1-Q4_K_M - 10
Llama-3.1-8B-Instruct-UD-Q4_K_XL - 11
Qwen3-8B-128K-UD-Q4_K_XL - 9
gemma-3-12b-it-Q6_K - 6
gemma-3-12b-it-UD-Q4_K_XL - 7
Mistral-Nemo-Instruct-2407-IQ4_XS - 10

Huihui-Ling-mini-2.0-abliterated-MXFP4_MOE - 58
inclusionAI_Ling-mini-2.0-Q6_K_L - 47
LFM2-8B-A1B-UD-Q4_K_XL - 38
ai-sage_GigaChat3-10B-A1.8B-Q4_K_M - 34
Ling-lite-1.5-2507-MXFP4_MOE - 31
granite-4.0-h-tiny-UD-Q4_K_XL - 29
granite-4.0-h-small-IQ4_XS - 9
gemma-3n-E2B-it-UD-Q4_K_XL - 28
gemma-3n-E4B-it-UD-Q4_K_XL - 13
kanana-1.5-15.7b-a3b-instruct-i1-MXFP4_MOE - 24
ERNIE-4.5-21B-A3B-PT-IQ4_XS - 28
SmallThinker-21BA3B-Instruct-IQ4_XS - 26
Phi-mini-MoE-instruct-Q8_0 - 25
Qwen3-30B-A3B-IQ4_XS - 27
gpt-oss-20b-mxfp4 - 23

So it seems I would get 3-4X performance if I build a desktop with 128GB DDR5 RAM 6000-6600. For example, above t/s * 4 for 128GB (32GB * 4). And 256GB could give 7-8X and so on. Of course I'm aware of context of models here.

Qwen3-4B-Instruct-2507-UD-Q8_K_XL - 52 (13 * 4)
gpt-oss-20b-mxfp4 - 92 (23 * 4)
Qwen3-8B-128K-UD-Q4_K_XL - 36 (9 * 4)
gemma-3-12b-it-UD-Q4_K_XL - 28 (7 * 4)

I stopped bothering 12+B Dense models since Q4 of 12B Dense models itself bleeding tokens in single digits(Ex: Gemma3-12B just 7 t/s). But I really want to know the CPU-only performance of 12+B Dense models so it could help me deciding to get how much RAM needed for expected t/s. Sharing list for reference, it would be great if someone shares stats of these models. 

Seed-OSS-36B-Instruct-GGUF
Mistral-Small-3.2-24B-Instruct-2506-GGUF
Devstral-Small-2507-GGUF
Magistral-Small-2509-GGUF
phi-4-gguf
RekaAI_reka-flash-3.1-GGUF
NVIDIA-Nemotron-Nano-9B-v2-GGUF
NVIDIA-Nemotron-Nano-12B-v2-GGUF
GLM-Z1-32B-0414-GGUF
Llama-3_3-Nemotron-Super-49B-v1_5-GGUF
Qwen3-14B-GGUF
Qwen3-32B-GGUF
NousResearch_Hermes-4-14B-GGUF
gemma-3-12b-it-GGUF
gemma-3-27b-it-GGUF

Please share your stats with your config(Total RAM, RAM Type - MT/s, Total Bandwidth) & whatever models(Quant, t/s) you tried.

And let me know if any changes needed in my llama-bench command to get better t/s. Hope there are few. Thanks