Has anyone here successfully extended Qwen3.5 or 3.6 context length paste 260k?

Posted by fragment_me@reddit | LocalLLaMA | View on Reddit | 11 comments

I've read about YARN, but I'm I'm not familiar with it. And this doesn't seem to work for me, the cap is still 260k. llama-server \ -m /path/to/qwen3.5-35B.gguf \ -c 400000 \ --rope-scaling yarn \ --rope-scale 1.526 \ --yarn-orig-ctx 262144

Reply to Post

11 Comments

[-]

FoxiPanda@reddit

So disclaimer: the answer to your question in the title for me is: no. However, looking through the documentation a bit, I think one thing here is that your -c parameter should be 262144 and then the --rope-scale is what acts as a multiplier and so you never really *see* the 400K context in your command, but you can infer it by doing the 262144*1.526 = 400031~ As for the 'yarn-orig-ctx' I couldn't figure out how to make that determination in my 5 minutes of looking... you may end up having to look through the llama.cpp launcher without -c set to try and figure this out? Not sure tbh. You may also want to look at this thread as I think there may be some additional settings you may need to mess with (i.e. --override-kv <insert architecture here>.context_length=int:16384) - see this thread for a few more hints: https://github.com/ggml-org/llama.cpp/issues/17459

[-]

fragment_me@reddit (OP)

Thanks for looking into it! After testing, this ended up working for me. I'm not testing some benchmarks. --ctx-size 300000 \ --rope-scaling yarn --rope-scale 1.14441 --yarn-orig-ctx 262144 \ --override-kv qwen35.context_length=int:1000000

[-]

FoxiPanda@reddit

Very cool, I'll add this to my notes to mess with. I am curious about how context rot affects any workload you might be running at this length, but that is... notoriously difficult to measure unless the model just completely falls apart.

[-]

fragment_me@reddit (OP)

I have been testing it for about an hour now. KV cache at BF16, and at 300k context it's still able to find what it needs. I basically gave it a TON of Rust source code files for the latest version of the Bevy game engine, and then I asked some questions about the files. The model (Qwen 3.6 35B) was able to retrieve it all so far. With 2x 3090 running with pipeline parallelism (-sm layer), and at 330k cache size, the pp tok/s is 412-470 and the tg tok/s is 24. At 50k context it was 1.3k pp tok/s and 54 tg tok/s. Another thing I noticed, you definitely need a prompt cache of more than 32GB of RAM for more than 250k context or else the prompt reprocessing will take a LONG time. I think 64GB should be fine for 500k context. As an aside, I used BF16 because apparently the model was trained on it, and I figured I shouldn't quant KV cache at this huge context. I also ran some benchmarks on BF16 vs F16 perplexity and KLD, which showed some minor but outside the margin of error differences.

[-]

FoxiPanda@reddit

It's pretty cool that it works at all though and doesn't completely fall apart at high contexts. Given your VRAM, I would be curious if you could grab a TurboQuant-enabled fork of Llama.cpp, build it, change the caching to something like --cache-type-k turbo3 & --cache-type-v turbo3 and re-run that same test using turboquant, but you're already layering experimental-ish features together, so I imagine eventually you'd hit some sort of bug that would derail things.

[-]

fragment_me@reddit (OP)

I can give it a shot, but I'm not optimistic about the Turbo Quant forks. The perplexity and KLD benchmarks in TheTom's fork seemed to be just slightly worse than Q4\_0. Given that, I may as well just run Q4\_0. I think the lowest one should go is Q8\_0 K and Q4\_0 V, but given this is high context, BF16/F16 K and Q8 V may be the sweet spot. Regardless, I'll try the TurboQuant fork to see what happens tonight.

[-]

Has anyone here successfully extended Qwen3.5 or 3.6 context length paste 260k?

Reply to Post

11 Comments

FoxiPanda@reddit

fragment_me@reddit (OP)

FoxiPanda@reddit

fragment_me@reddit (OP)

FoxiPanda@reddit

fragment_me@reddit (OP)

FoxiPanda@reddit

Farmadupe@reddit

cafonez@reddit

Ok_Warning2146@reddit

putrasherni@reddit