I run this model that is downloaded and setup by this install.sh

https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4

with this shutdown/startup script, and vllm options that I'm experimenting with:

docker stop vllm-qwen35
docker rm vllm-qwen35
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
docker run -d --name vllm-qwen35 \
  --gpus all --net=host --ipc=host \
  -v ~/models:/models \
  vllm-qwen35-v2 \
  serve /models/qwen35-122b-hybrid-int4fp8 \
  --served-model-name qwen \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.90 \
  --port 8000 \
  --host 0.0.0.0 \
  --load-format fastsafetensors \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --generation-config auto \
  --override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "presence_penalty": 0.0, "repetition_penalty": 1.0}'

I just slapped this together last night from various sources. vLLM is poorly documented in my experience and there can be laughable problems with the above.

I think --max-num-seqs should be set lower from whatever its default is, which I suspect to be 16. For hardware like GB10, that is needlessly many -- the thing barely has the power to run about 4 inference jobs in parallel without running out of compute, and I suspect it's more like 2-3 when you have speculative config set. What I'm certain of is that the capacity for large number of parallel sequences eats up your VRAM.

The --load-format fastsafetensors is also essential, as it drops model loading time by like 80 %. It is just ridiculously slow without this. Even with this setting, starting vllm takes like 10 minutes...

Chunked prefill should speed up prompt processing, though I have no data if that is the case, the auto tool choice, tool call parser and reasoning parser are set like this because this works in opencode. I know the git repo said I should use qwen3_xml but tool calls immediately stopped working in opencode, so I don't think so. I don't know what is up with that. Getting rid of reasoning parser also will confuse the model greatly, so it has to be there. There is apparently a bug between reasoning and tool call parser in that the reasoning parser runs first and extracts the think sequence out, and the tool call parser runs on what remains, and if the model wrote tool call into think section, it misses it.

The generation config and override is just my attempt to guarantee that sampling is being performed with qwen3.5 recommendations for coding. I have no idea how to confirm what settings are used, as vllm really doesn't like to print this type of information that would help me to ascertain the correctness of the parameters that inference is executing under.

Anyway, prompt is around 1200 tok/s and generation around 50 tok/s with this repo. I'm seeing variation from 40-60 on medium sized prompts, very short completions are < 10 tok/s, likely some overhead causes such a low figure. I am not sure if the model is at full quality as this bastardized int4/fp8 combo, but it seems to behave quite well and while I think there could be slight gap to what I'm used to getting from 6-bit GGUF inference, the speed more than makes up for it.