qwen3.5-122b-a10b-fp4:
    env:
      - VLLM_LOG_STATS_INTERVAL=5
      - VLLM_SKIP_P2P_CHECK=1
      - VLLM_ENABLE_PCIE_ALLREDUCE=1
      - NCCL_P2P_LEVEL=SYS
      - CUDA_DEVICE_ORDER=PCI_BUS_ID
      - CUDA_VISIBLE_DEVICES=5,4,3,2,1,0
      - OMP_NUM_THREADS=12
      - VIRTUAL_ENV=/home/gleb/llm/env_qwen_35
    cmd: |
      /home/gleb/.local/bin/uv run
        -m vllm.entrypoints.openai.api_server
        #--model /mnt/samsung_990_evo/llm-data/models/Qwen/Qwen3.5-122B-A10B-GPTQ-Int4
        #--model /mnt/samsung_990_evo/llm-data/models/Sehyo/Qwen3.5-122B-A10B-NVFP4
        #--model /mnt/samsung_990_evo/llm-data/models/Intel/Qwen3.5-122B-A10B-int4-AutoRound
        --model /mnt/samsung_990_evo/llm-data/models/cyankiwi/Qwen3.5-122B-A10B-AWQ-4bit
        #--model /mnt/samsung_990_evo/llm-data/models/happypatrick/Qwen3.5-122B-A10B-heretic-int4-AutoRound
        --served-model-name "qwen3.5-122b-a10b-fp4"
        --port ${PORT}
        --tensor-parallel-size 4
        --pipeline-parallel-size 1
        --max-num-batched-tokens 4096
        --enable-prefix-caching
        --max-model-len auto
        --gpu-memory-utilization 0.955
        --max-num-seqs 2
        --attention-backend flashinfer
        --reasoning-parser qwen3
        --enable-auto-tool-choice
        --tool-call-parser qwen3_coder
        --load-format instanttensor
        #--enable-expert-parallel
        #--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":5}'

Key points: - All GPUs have 0Gb reserved by system, all 24Gb are free. No GUI, no DE, no WM - only terminal under ubuntu server. - --max-num-seqs 2 - default value will try to capture graphs for 16 users - too much for 96Gb VRAM. - --attention-backend flashinfer - free performance! - --max-num-batched-tokens 4096 - default is 2048, but 4096 is faster PP. - --gpu-memory-utilization 0.955 - ONLY achievable if 0Gb reserved by system - after weights loaded I have Available KV cache memory: 2.64 GiB in logs, then Auto-fit max_model_len: reduced from 262144 to 223872 to fit in available GPU memory (2.64 GiB available for KV cache). For example, CachyOS and KDE uses 2.677Gi/31.843Gi on my 5090. OS uses more VRAM than is available for KV cache. There are ZERO chances you will be able to run this model shared with OS desktop.