MY advice based on my own experience running LLMs that have to spill into system ram and run some layers on GPUs and others on CPU:

Run on linux (not windows, not WSL, actual linux server).

Do not run it at anything less than q4 (at q2 and less you're just pissing away electricity on a nerfed model and at that point you'd get better performance using higher quant of a smaller model).

Use llama-server to host the model (Llama CPP). No not Ollama, it's slower and has fewer options.

Use Claude Code with full access to your linux server to help you test different llama-server configurations and context window sized so you can optimally offload MOE layers on CPU vs GPUs. You can ask it to run a few benchmarks at different context sizes and have some options to choose from.

Obviously don't copy these settings because this is for my hardware and a totally different model, but this was what Claude Code and I figured out iteratively was the best configuration to run Qwen 35b on my specific hardware (involving some spill over into system ram and CPU run model layers):

```
-m ~/models/Qwen3.5-35B-A3B-Q4_K_M.gguf
--mmproj ~/models/mmproj-F16.gguf     # vision projector
--no-mmproj-offload                    # keep vision projector in RAM, not GPU
--host 0.0.0.0 --port 8080
-ngl 999                               # offload all layers to GPU
--flash-attn on
--split-mode layer --tensor-split 8,12  # split across GPUs proportional to VRAM
--n-cpu-moe 23                         # 23 of 40 MoE layers'experts offloaded to CPU RAM
--kv-unified
--cache-type-k bf16 --cache-type-v bf16 # bf16 KV cache (avoids gibberish)
--batch-size 4096 --ubatch-size 1024
--ctx-size 65536                        # 65K context window
--parallel 1                            # single request slot (see concurrency section)
--jinja                                 # Jinja2 chat templates
```