I was also able to replicate the results. with caveats

# vLLM Optimization Findings


## Hardware
- Dual RTX 3090 GPUs (24GB VRAM each)
- Model: cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4


## Current Best Configuration
```yaml
--tensor-parallel-size 2
--gpu-memory-utilization 0.97
--max-model-len 131072
--quantization compressed-tensors
--max-num-seqs 8
--block-size 32
--max-num-batched-tokens 2048
--enable-prefix-caching
--attention-backend FLASHINFER
--speculative-config '{"method":"mtp","num_speculative_tokens":5}'
--compilation-config '{"cudagraph_mode": "PIECEWISE"}'
-O3
```


## Key Findings


### 1. CUDA_LAUNCH_BLOCKING
- 
**Finding**
: Significantly slows down inference (2-3x slowdown)
- 
**Action**
: Remove this flag


### 2. Attention Backend: FLASHINFER vs FLASH_ATTN
- 
**Finding**
: FLASHINFER is faster for AWQ/int4 models (50-96 tok/s vs 20-68 tok/s)
- 
**Action**
: Use `--attention-backend FLASHINFER`


### 3. Speculative Decoding (MTP)
- 
**Finding**
: Performance depends heavily on acceptance rate
- 
**Coding prompts**
: ~80% acceptance → 106 tok/s
- 
**Creative writing**
: ~27-49% acceptance → 37-71 tok/s
- 
**Action**
: Works best with predictable/structured tasks (coding)


### 4. GPU Memory Utilization
- 
**Finding**
: 0.98 causes OOMs, 0.97 is stable, 0.96 is safe
- 
**Action**
: Use 0.97 for best balance of performance and stability


### 5. CUDAGraph Mode
- 
**Finding**
: PIECEWISE required for spec-decoding + FLASHINFER
- 
**Action**
: Use `--compilation-config '{"cudagraph_mode": "PIECEWISE"}'`


### 6. -O3 Optimization Level
- 
**Finding**
: Enables VLLM_COMPILE mode (mode: 3)
- 
**Action**
: Keep `-O3` flag


### 7. Open WebUI Timeout
- 
**Finding**
: Caused by Nginx Proxy Manager timing out before streaming completes
- 
**Action**
: Add to Nginx config:
  - `proxy_read_timeout 300s;`
  - `proxy_send_timeout 300s;`
  - `proxy_buffering off;`


## Benchmark Results


| Prompt Type | Throughput | Acceptance Rate |
|-------------|-----------|-----------------|
| Coding (Python function) | 106 tok/s | 80.5% |
| Creative story | 37-71 tok/s | 26-49% |


### Test Prompts


**Slow (Creative Writing):**
```
Write a creative story about a person who discovers they can communicate with animals.
```


**Fast (Coding):**
```
Write a Python function that takes a list of dictionaries containing user data (name, email, age) and returns a new list with only users over 18, sorted by age. Include type hints, docstrings, and error handling.
```


## Notes
- Prefix caching improves subsequent request performance (0% → 73%+ hit rate)
- spec-decoding overhead not worth it for low-acceptance scenarios
- Model architecture: Qwen3_5ForConditionalGeneration with MTP drafter

services:
  vllm:
    image: vllm/vllm-openai:latest-cu130
    container_name: vllm
    restart: unless-stopped
    # ports:
    #   - "8999:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      # - VLLM_LOGGING_LEVEL=DEBUG
      # - VLLM_LOG_STATS_INTERVAL=1
      # - NCCL_DEBUG=TRACE
      # - VLLM_TRACE_FUNCTION=1
      # - NCCL_IGNORE_DISABLED_P2P=1
      # - CUDA_LAUNCH_BLOCKING=1
      - VLLM_API_KEY=[YOUR_KEY_HERE]
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
      - CUDA_VISIBLE_DEVICES=0,1
      - RAY_memory_monitor_refresh_ms=0
      - NCCL_CUMEM_ENABLE=0
      # - VLLM_SLEEP_WHEN_IDLE=1
      - VLLM_ENABLE_CUDAGRAPH_GC=1
      - VLLM_USE_FLASHINFER_SAMPLER=1
    shm_size: 4g
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]
    command: >
      cyankiwi/Qwen3.5-27B-AWQ-BF16-INT4
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.97
      --max-model-len 131072
      --quantization compressed-tensors
      --max-num-seqs 8
      --block-size 32
      --max-num-batched-tokens 2048
      --enable-prefix-caching
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --reasoning-parser qwen3
      --attention-backend FLASHINFER
      --speculative-config '{"method":"mtp","num_speculative_tokens":5}'
      --compilation-config '{"cudagraph_mode": "PIECEWISE"}'
      --no-use-tqdm-on-load
      -O3
    networks:
      - reverse-proxy-net

networks:
  reverse-proxy-net:
    name: reverse-proxy-net
    external: true