I use several models for different things; I run almost exclusively llama.cpp, and I use llama-swap to sit in front of my llama-server instances, providing around 32 different model choices. (I test different models regularly.) My go-to has been MoE models since \~GLM-4.6, as I can split them between GPU and CPU, and they handle it much better than dense models.

Right now I'm using GLM-5.1 (unsloth/GLM-5.1-GGUF)at 3 bit quantization, generally in non-reasoning mode, for creative writing. It's also my go-to for anything where I want to talk, but not do tool calls. At 10 t/s generation local, it's just way too slow at that, but it's human-speed for conversation or character-driven stories. It also picks up on character-definitions better than any other model short of Opus. (I've also used GLM-5.1 in the cloud for OpenClaw historically, because it's Opus-level smart, and because I want my agent to adopt the persona that is defined for it. These days I'm trying to use Qwen-122B local more consistently for OpenClaw, unless I need the smarts.)

For agentic use, Qwen 3.5-122B works surprisingly well, although not much of a 'personality'. I've run it at q4 (fully in GPU RAM) at \~50 t/s generation. I haven't needed to push up to q8 for it, and if I need much smarter I go cloud. Now the specific model I'm using there is HauhauCS/Qwen3.5-122B-A10B-Uncensored-HauhauCS-Aggressive. I also use that for image analysis, tagging and processing using mmproj-f16. The q4 isn't as good at image analysis as the q8, though. If I had to pick a model to stick in pure GPU RAM semi-permanently, it'd probably be this one, although I'd bump up to Q8 and let some of it sit in RAM.

I have an embedding model, but I don't really use it that much anymore. I was using Qwen3-Embedding-8B-Q8_0.gguf and a smaller Qwen reranker. I need to get back to this.

My system is a ASUS ESC4000A-E12 with a 32 core EPYC and 384GB of DDR5 RAM, 2xL40S for 96GB of GPU RAM; it sits in a SysRacks rack in my garage.

My basic config for each llama.cpp llama-server call in the llama-swap config expands to:

llama-server --prio 2 --mmap --log-timestamps --kv-unified --fit on --metrics \
  --jinja --temp 1.0 --min-p 0.01 --top-p 0.95 --threads-http 8 --mclock \
  --host 0.0.0.0 --port ${PORT} \
  --flash-attn on -ctk q8_0 -ctv q8_0

For non-reasoning, I add:

--reasoning-budget 0 --chat-template-kwargs '{"enable_thinking": false}'

I customize the -c {context-length} per model, because if you don't manually set a context length, --fit on will shrink your context to nothing, in order to fit the models, before it goes to RAM. :rage:

I also have a 'limited reasoning' for use cases where I want it to do reasoning, but I want it to not waste all it's time doing it. So I'll limit it to \~2048 tokens of reasoning, and leaving enable_thinking alone. (E.g. using the above 50 t/s on Qwen3.5-122B@Q4, that's about 41 seconds of reasoning.)

Hope that helps!