Getting \~43 narrative / \~54 code TPS at 330W on a single RTX 3090 with fp8 KV + MTP n=3. Reference setup (identical config, same GPU model) claims 66/84 TPS. MTP acceptance rates are comparable or better (93/87/74% vs 92/81/64%), but base decode throughput is \~20 TPS lower. Looking for ideas on what's causing the gap.

Full Configuration

Docker Image

vllm/vllm-openai@sha256:9bba4628a3b943e0dd33caefb94b811569ba1e97bdf23bee19a265c31b947ccb
# v0.19.2rc1.dev21+g893611813

vLLM Launch Args

--model /root/.cache/huggingface/vllm-qwen36-27b-int4
--served-model-name qwen3.6-27b-autoround
--quantization auto_round
--dtype float16
--tensor-parallel-size 1
--max-model-len 75000
--gpu-memory-utilization 0.97
--max-num-seqs 1
--max-num-batched-tokens 2048
--kv-cache-dtype fp8_e5m2
--language-model-only
--trust-remote-code
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--enable-prefix-caching
--enable-chunked-prefill
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
--host 0.0.0.0 --port 8000

Environment Variables

VLLM_WORKER_MULTIPROC_METHOD=spawn
NCCL_CUMEM_ENABLE=0
NCCL_P2P_DISABLE=1
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
VLLM_NO_USAGE_STATS=1
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512
VLLM_FLOAT32_MATMUL_PRECISION=high
VLLM_USE_FLASHINFER_SAMPLER=1
OMP_NUM_THREADS=1
CUDA_DEVICE_MAX_CONNECTIONS=8
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
VLLM_MARLIN_USE_ATOMIC_ADD=1

Patches (applied before vLLM start)

1. Genesis v5.10 — 20/21 applied, 0 failed (1 already done)
2. Key patches: Marlin FP8 fallback, TQ hybrid support, MoE fast path, Qwen3 tool_call fix, mamba .get() guard, TQ decode stage1 tune, TQ prealloc dequant+cu
3. tolist_cudagraph_fix (from noonghunna/qwen36-27b-single-3090) — Site A: ok, Site B: ok
4. Wraps .tolist() calls in turboquant_attn.py with torch.cuda.is_current_stream_capturing() guards so CUDA graph capture doesn't crash

Model

Lorbus/Qwen3.6-27B-int4-AutoRound
- Architecture: Qwen3_5ForConditionalGeneration (Qwen3.6 hybrid: 48 linear_attn + 16 full_attn)
- Quantization: AutoRound INT4, mtp.fc preserved as BF16
- Size: ~18 GiB on disk, 16.87 GiB in VRAM
- MTP: Qwen3_5MTP, shares embedding + lm_head with target model

Runtime Details

vLLM version:              0.19.2rc1.dev21+g893611813
Architecture resolved:      Qwen3_5MTP
Quantization backend:      inc (AutoRound INT4)
Weight kernel:              MarlinLinearKernel for GPTQMarlinLinearMethod
Attention backend:          FlashInfer
CUDA graphs:                FULL_AND_PIECEWISE → downgraded to PIECEWISE
  WARNING: CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode
            for FlashInferBackend (UNIFORM_SINGLE_TOKEN_DECODE);
            setting cudagraph_mode=PIEWISE
torch.compile:              53s (backbone) + 12s (eagle_head), cached to disk
CUDA graph capture:         4 graphs (sizes 1,2,4,8), 0.08 GiB
KV cache:                   fp8_e5m2, ~21.3% GPU usage at 1K context
Model VRAM:                  16.87 GiB
Total VRAM used:            ~22.1 GiB / 24.5 GiB
MTP:                         n=3, shared embedding + lm_head
Driver:                      580.119.02, CUDA 13.0
GPU:                         RTX 3090 (Ampere SM_86)
Power cap:                   330W (stock is 230W)

Key Warning

CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for
attention backend FlashInferBackend (support: UNIFORM_SINGLE_TOKEN_DECODE);
setting cudagraph_mode=PIECEWISE

Benchmark Results

At 330W Power Cap (after 3 warmup rounds)

Narrative (800-word essay, max_tokens=1000, temp=0.6):
  narr1:  45.1 TPS
  narr2:  45.7 TPS
  narr3:  44.3 TPS

Code (quicksort with comments, max_tokens=800, temp=0.6):
  code1:  58.6 TPS
  code2:  57.3 TPS

At 230W Power Cap (stock, after warmup)

Narrative:  ~33 TPS
Code:       ~40 TPS

MTP SpecDecoding Metrics (330W, warm)

Per-position acceptance:  93.0% / 86.6% / 73.9%
Mean acceptance length:     3.55
Avg draft acceptance rate:  85.1%
Drafted throughput:         ~48 tok/s
Accepted throughput:         ~34-37 tok/s
GPU usage:                   21.3% KV, ~90% compute
Power draw:                  ~250W

Reference Comparison

Things I've Already Checked

• Marlin kernels active — Using MarlinLinearKernel for GPTQMarlinLinearMethod
• CUDA graphs working — PIECEWISE mode, NOT enforce-eager
• Genesis patches all passing — 20/21 applied, 0 failed
• tolist cudagraph patch applied — both sites patched
• MTP sharing weights — Detected MTP model. Sharing target model embedding/lm_head weights with the draft model.
• fp8_e5m2 KV — NOT turboquant (turboquant+spec-decode broken per vllm#40831)
• language-model-only — no vision tower loaded
• Model is correct Lorbus INT4 — mtp.fc.weight present as BF16 (not quantized)
• torch.compile caching — compiled and cached to disk (53s backbone, 12s eagle)
• Power cap — tested at 330W (+10% TPS vs 230W)
• VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 — enabled
• VLLM_FLOAT32_MATMUL_PRECISION=high — set
• No enforce-eager — CUDA graphs are active

Potential Causes I'm Unsure About

1. CUDA graphs downgraded to PIECEWISE — The FlashInfer backend doesn't support FULL_AND_PIECEWISE with spec-decode. Is the reference also running PIECEWISE, or did they get FULL mode working somehow?
2. Draft throughput bottleneck — My drafted throughput is only \~48 tok/s. If base decode is the bottleneck, acceptance rate improvements don't help much. What drafted throughput should I expect?
3. torch.compile cache persistence — The compile cache is inside the Docker container at /root/.cache/vllm/torch_compile_cache/. Not mounted as a volume, so it rebuilds on restart. Could this affect warm-run performance?
4. Model path vs HuggingFace repo name — My model is loaded from a local directory /root/.cache/huggingface/vllm-qwen36-27b-int4 rather than the HuggingFace repo name Lorbus/Qwen3.6-27B-int4-AutoRound. Could this affect any auto-configuration?
5. Genesis patch version — I'm running v5.10, the repo now has v7.10 (which uses a plugin architecture). Could newer patches improve TPS?
6. max_num_batched_tokens=2048 — vLLM warns this is suboptimal with spec-decode. The reference uses the same value but could there be a better setting?

Docker Compose (Complete)

services:
  vllm-qwen36-27b:
    image: vllm/vllm-openai@sha256:9bba4628a3b943e0dd33caefb94b811569ba1e97bdf23bee19a265c31b947ccb
    container_name: vllm-qwen36-27b
    restart: "no"
    ports:
      - "8020:8000"
    volumes:
      - /run/media/will/Storage/models:/root/.cache/huggingface
      - /home/will/genesis-vllm-patches/patch_genesis_unified.py:/patches/patch_genesis_unified.py:ro
      - /home/will/genesis-vllm-patches/patch_tolist_cudagraph.py:/patches/patch_tolist_cudagraph.py:ro
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN:-}
      - VLLM_WORKER_MULTIPROC_METHOD=spawn
      - NCCL_CUMEM_ENABLE=0
      - NCCL_P2P_DISABLE=1
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
      - VLLM_NO_USAGE_STATS=1
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb=512
      - VLLM_FLOAT32_MATMUL_PRECISION=high
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - OMP_NUM_THREADS=1
      - CUDA_DEVICE_MAX_CONNECTIONS=8
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - VLLM_MARLIN_USE_ATOMIC_ADD=1
    shm_size: "16gb"
    ipc: host
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    entrypoint:
      - /bin/bash
      - -c
      - |
        set -e
        pip install xxhash -q
        python3 /patches/patch_genesis_unified.py
        python3 /patches/patch_tolist_cudagraph.py
        exec vllm serve "$@"
      - --
    command:
      - --model
      - /root/.cache/huggingface/vllm-qwen36-27b-int4
      - --served-model-name
      - qwen3.6-27b-autoround
      - --quantization
      - auto_round
      - --dtype
      - float16
      - --tensor-parallel-size
      - "1"
      - --max-model-len
      - "75000"
      - --gpu-memory-utilization
      - "0.97"
      - --max-num-seqs
      - "1"
      - --max-num-batched-tokens
      - "2048"
      - --kv-cache-dtype
      - fp8_e5m2
      - --language-model-only
      - --trust-remote-code
      - --reasoning-parser
      - qwen3
      - --enable-auto-tool-choice
      - --tool-call-parser
      - qwen3_coder
      - --enable-prefix-caching
      - --enable-chunked-prefill
      - --speculative-config
      - '{"method":"mtp","num_speculative_tokens":3}'
      - --host
      - 0.0.0.0
      - --port
      - "8000"