27B Local Inference on Single RTX 3090

qwen3.6-27B-AutoRound (INT4), vLLM 0.19.2rc1.dev21, 24GB VRAM.

71–83 tok/s after warmup.

• Turboquant 3-bit NC KV Cache: Compresses KV state to 3-bit non-uniform quantization. Enables 125K context window within 24GB VRAM without OOM. • MTP n=3 Speculative Decoding: Three auxiliary heads draft tokens per forward pass, verified atomically against main head. ~3× throughput multiplier vs. non-speculative baselines. • Cudagraph PIECEWISE Mode: Captures only attention-op boundaries instead of full-graph replay. Eliminates degenerate repetition loops caused by stale MTP state in FULL_AND_PIECEWISE mode on multi-GPU hosts. • Chunked Prefill + Prefix Caching: max-num-batched-tokens=4121 with max-num-seqs=1. First post-restart request incurs ~29s cudagraph compilation; subsequent requests stabilize at 12–14s for 1024-token generation.

I've been spending too much time staring at GPU util graphs, but here's the thing: running a 27B local model on a single consumer GPU in 2026 is like trying to cook a ten-course tasting menu in a one-burner kitchen. You don't need a bigger kitchen. You need to stop being wasteful.

The rig: RTX 3090 (24GB), AMD 9900X, 192GB DDR5. Nothing exotic. The kind of box a mid-tier game dev would run on. The model is qwen3.6-27B-AutoRound (INT4 quantized), served by vLLM.

Here are the tricks that make this actually work instead of choking at 12 tokens a second:

TURBOQUANT 3-BIT KV Cache. This is the move nobody talks about. Instead of stashing every attention computation in full 16-bit precision (the safe, default choice), we compress the KV cache to 3-bit. It's like storing your recipes on cocktail napkins instead of index cards. You think you'll lose something. You don't. On a 3090 with 24GB, this is the difference between "fits" and "OOM-killed by your own ambition."

MTP — Multi-Token Prediction. vLLM speculates the next three tokens using auxiliary heads, then verifies them in one pass against the main model. It's like hiring three sous-chefs to prep ingredients while you plate. When it works — and this is the crucial part, when it works — you get roughly triple the throughput because three speculative tokens get accepted per forward pass instead of one. We're seeing 71 to 83 tokens per second. That's not "usable." That's fast.

Cudagraph mode PIECEWISE, not FULL. This was the trap. The default FULL_AND_PIECEWISE captures complete execution graphs and replays them. On this machine, with a 1660 Ti also plugged in (legacy display adapter, I know, don't look at me like that), FULL capture poisoned the speculative decoding. The model started outputting the same token in a loop — 100% MTP acceptance, zero intelligence. Switched to PIECEWISE, which only captures attention-operation boundaries. No more garble. No more repetition loops. Just clean, fast inference.

The warmup tax: First request after a restart takes \~43 seconds for 1000 tokens because cudagraphs compile on the fly. After that? 14 seconds. Subsequent runs? 12 seconds. The kitchen warms up. Don't panic.

The first time I sat down at my mother-in-law's stove and realized I didn't need a perfect recipe, just a good one — that's this feeling. You don't need an H100. You need to stop being a tourist.