Haven't tried CPU inference for a while and back then (6 months?) it was painfully slow, interesting to see these MoE models running sort-of acceptably well on CPU.

Did a full bench sweep (custom self-improving script generated with Claude Code/Opus 4.7) on Gemma 4 26B-A4B Q4_K_XL on an i7-14700K + 96GB DDR5, CPU only via llama.cpp Docker image.

Real-world server numbers (warmed up, \~200 tok prompt → 300 tok gen):

Prompt Processing (PP): \~90 tok/s

Token Generation (TG): \~13 tok/s

Bench notes:

1. TG is bandwidth-bound and peaks at 8 threads (one per P-core, no HT). PP is compute-bound and keeps scaling all the way to 28 threads (using the slower E-cores). Use --threads 8 --threads-batch 28 in llama-server and you get both peaks from the same process. Setting threads=8 for everything caps PP at \~73; threads=28 for everything tanks TG to \~11. For short interactive prompts, forcing to P-cores might be worthwhile but not worth it for longer or background tasks.
2. docker --cpuset-cpus=0-15 to force everything onto P-cores looked great in synthetic bench (80 PP / 14.5 TG) — but in real serving PP collapsed to 44 tok/s. OpenMP HT contention under HTTP + sampling load. So llama-bench numbers don't always translate to live serving.
3. Stuff that didn't do much: mmap on/off, ubatch 256/512/1024 (within noise; <256 hurts), Flash Attn \~+2%. KV cache: stick with f16 unless you also use -fa 1 (q8_0/q4_0 KV refuse to load without flash-attn).
4. Using btop (beautiful TUI) this didn't seem to max out my full CPU, just individual cores. Surprisingly there wasn't much temperature spiking.

For OP on i5-8500 + DDR4: expect roughly half the TG (\~6-7 tok/s) since dual-channel DDR4 is \~40 GB/s vs DDR5's \~80, and PP will be lower again because of fewer cores. Would need \~22GB to load this into memory and have 128K context. Still very usable for an "almost-27B" model.

My GPU isn't the hottest so wondering if this would be a good way to run a second (or more) model at the same time, so my background batch jobs where I don't care about speed can use the CPU during the hours when I am actively using the GPU.

Here's my serve-cpu.sh. This is tuned for my CPU, might need tweaking for other setups:

#!/usr/bin/env bash
#
# CPU-only serving config for Gemma 4 26B-A4B-it UD-Q4_K_XL
# MoE: 26B total / 3.8B active — well-suited to CPU inference
# Hardware: i7-14700K (8 P-cores w/HT = CPUs 0-15, 12 E-cores = CPUs 16-27)
#
# Benchmark + real-server findings (2026-05-05):
#   - Asymmetric thread counts win: TG is bandwidth-bound (peaks at ~8 threads),
#     PP is compute-bound (scales to all 28). --threads 8 --threads-batch 28
#     gives both peaks via the same process.
#   - fa=on:      +2% PP, +2.5% TG. Also enables quantized KV (which needs FA).
#   - ub=512:     optimal. PP collapses below 256.
#   - f16 KV:     no reason to quantize — 96GB RAM available.
#   - mmap:       no measurable difference with mlock active.
#   - ctx scale:  PP at 8K ≈ 59 tok/s (-20% vs 512). Linear-ish degradation.
#
# Real-server measurement (198-tok prompt → 300-tok gen, warmed up):
#   PP: 88-96 tok/s   TG: 12.4-13.7 tok/s
# 
# Pinning experiments (do NOT add these — they make things worse here):
#   - llama.cpp's --cpu-mask / --cpu-strict are SILENTLY IGNORED by this build
#     (OPENMP=1; OpenMP runs its own thread pool). Verified by inspecting
#     /proc/<pid>/task/*/status — every thread shows Cpus_allowed_list: 0-27
#     regardless of --cpu-mask setting.
#   - docker --cpuset-cpus=0-15 with -tb 16 caused PP to collapse to ~44 tok/s
#     in real serving (vs 80 in synthetic bench). HT contention on P-cores
#     under HTTP+sampling load. Not worth the +13% TG.
# 
# vs GPU (RTX 4000 SFF Ada, serve.sh): TG 60 → ~13 tok/s (~4.5× slower).
# Viable for batch / offline; painful for interactive chat.

CONTEXT=32768
MODEL=gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf
MODEL_PATH=models/unsloth/gemma-4-26B-A4B-it-GGUF
MMPROJ=mmproj-BF16.gguf

docker run \
  --ipc=host \
  --shm-size=16g \
  -v ../../../models:/models \
  -p 11456:8080 \
  ghcr.io/ggml-org/llama.cpp:full \
  --server -m /${MODEL_PATH}/${MODEL} \
  --mmproj /${MODEL_PATH}/${MMPROJ} \
  --host 0.0.0.0 \
  --port 8080 \
  --flash-attn on \
  --ctx-size $CONTEXT \
  --cont-batching \
  -b 2048 \
  -ub 512 \
  --threads 8 \
  --threads-batch 28 \
  --n-gpu-layers 0 \
  --no-mmap \
  --mlock \
  --metrics \
  -np 1 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --jinja \
  --chat-template-kwargs '{"enable_thinking":false}' \
  --temp 1.0 --top-p 0.95 --top-k 64