Hopefully that might be helpful (Codex Generated, based on logs and etc)

What Happens:

The model boots cleanly and accepts requests, but real OpenCode long-context requests fail during prefill, not during startup or decode.

Failure mode:

- Client sends a large OpenCode request/tool context.

- vLLM begins chunked prefill.

- CUDA OOM occurs inside the compiled Qwen FFN path.

- Server returns HTTP 500 / stream breaks, and the vLLM process shuts down.

The observed OOM site is:

/root/.cache/vllm/.../inductor_cache/...py", line 1208

buf9 = empty_strided_cuda((s18, 17408), (17408, 1), torch.float16)

torch.OutOfMemoryError: Tried to allocate 138.00 MiB

OOM numbers from the failing run:

GPU total: 23.56 GiB

Free at failure: 131.75 MiB

Process memory in use: 23.40 GiB

PyTorch allocated: 22.59 GiB

CUDA graph private pools: 24.00 MiB

PyTorch reserved but unallocated: 472.77 MiB

Requested allocation: 138.00 MiB

Input Size

The synthetic stress test uses about 100000 chars, roughly 25K tokens, as a mock tool response. That synthetic test succeeds on the current vllm-text profile.

For the real OpenCode failure, vLLM did not log the exact prompt token count in the crash trace. Based on the failure class, it is happening in the same region: large tool/context prefill \~30K-tokens OpenCode tool/schema/context.

Configs Tested

We tested the relevant long-context vLLM profiles and memory-util variants, including 0.975 and 0.985 + lower context lenghts; the real OpenCode workload still breaks. Lowering/raising mem-util shifts the KV/activation tradeoff but does not reliably create enough activation headroom for the real request.

The current synthetic test passes on:

vllm-qwen36-27b-long-text

max_model_len=128000

gpu_memory_utilization=0.975

language_model_only=True

Synthetic result:

\~25K-token mock tool prefill

HTTP 200

response length: 723 chars

finish=stop

Why It Fails

This is no longer the FA2 softmax_lse Cliff 1 path. The FA clamp sidecar and PN12 sidecar are both applying.

Confirmed applied:

[pn12_ffn_pool_anchor_fix] SiluAndMul.forward_cuda: applied

[fa_max_seqlen_clamp] _flash_attn_varlen: applied

The remaining failure is the FFN intermediate activation allocation:

empty_strided_cuda((s18, 17408), ..., torch.float16)

That buffer is roughly:

max_num_batched_tokens × intermediate_size × fp16

≈ 4128 × 17408 × 2 bytes

≈ 138-144 MiB

At the moment of failure, the GPU only has \~131 MiB free, so the allocation misses by a few MiB. Real OpenCode requests appear to create a worse activation/KV/cache pressure profile than the synthetic test, even when the synthetic 25K tool-prefill succeeds.

Yeah absolutely, latest commit.