Our nonprofit association has an AI server with 2x RTX 3090 and I finally switched over to vLLM to get better performance for multiple users.

Here's my docker compose file:

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - VLLM_API_KEY=my_very_secret_key_was_scrubbed
    volumes:
      - /opt/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "8000:8000"
    ipc: host # Prevents shared memory bottlenecks during tensor parallelism
    command: >
      --model cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
      --tensor-parallel-size 2
      --max-model-len 65536
      --gpu-memory-utilization 0.85
      --enable-prefix-caching
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --max-num-seqs 32
      --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
    restart: unless-stopped

I'm super happy with it, but if you have suggestions for improvements, let me know!

Here are my llama-benchy results: