Qwen3.5 35b UD Q4 K XL Prior to 3/5 worked great, now not so much...

Posted by thejacer@reddit | LocalLLaMA | View on Reddit | 19 comments

I committed a party foul and deleted my .gguf before testing the updated ones and now I'm stuck with loops and strange characters. Prior to 3/5 update UD Q4 K XL was great with just occasional loops and Chinese (handful of times in millions of tokens) but the UD Q6 K XL looped a lot. Saw the post about the update today so I deleted my file and downloaded the new one...RIP. Now the UD Q4 K XL is unusable, looping and printing weird characters in half my prompts. So I downloaded the Bartowski Q4 K L and it WORKS but it thinks about 50% more than the UD Q4 K XL (prior to 3/5). How are the updated quants working for everyone else?

Reply to Post

19 Comments

[-]

ShuraWW@reddit

The loops and weird characters after the 3/5 update sounds like a bad quant file or tokenizer mismatch. A few things to try: 1. Make sure you're using --jinja flag - without it, chat templates can tokenize inconsistently and cause garbage output. 2. Check if your llama.cpp version is compatible with the new quant. There were some PRs merged recently (>= build 8140) that fixed Qwen3.5 checkpoint issues. 3. Try adding --swa-full if you're not already using it. 4. Could also be a corrupted download - maybe redownload and verify the hash? The fact that Bartowski Q4\_K\_L works fine suggests it's something specific to the UD quant file, not your setup. Maybe they broke something in the 3/5 update. What flags are you running with?

[-]

hilycker@reddit

"srv load\_model: swa\_full is not supported by this model, it will be disabled" running llama.cpp [b8779](https://github.com/ggml-org/llama.cpp/releases/tag/b8779). I don't think swa is supported in the whole Qwen3.5 family, both MoE and dense models

[-]

thejacer@reddit (OP)

Bruh...I thought jinja enabled was default...trying with --jinja now. I was on 8189 (doing a rebuild) and here are my flags: \-m /models/Qwen3.5-35B-A3B-UD-Q6\_K\_XL.gguf \--host [0.0.0.0](http://0.0.0.0) \--port 8080 \-ngl 99 \--mmproj /models/mmproj\_Qwen3\_5-F16.gguf \--no-mmap \-fa on \--api-key key \--temp 1.0 \--top-p 0.95 \--top-k 20 \--min-p 0.0 \--presence-penalty 1.5 \--repeat-penalty 1.0

[-]

FluoroquinolonesKill@reddit

> thought jinja enabled was default According to the help file, it is indeed. I do not pass --jinja on my end.

[-]

thejacer@reddit (OP)

Maybe it just required a relaunch, but with the --jinja flag added I haven't gotten a single loop with the exact same files.

[-]

ShuraWW@reddit

Try adding these flags: \--jinja --swa-full --ctx-checkpoints 512 Also --no-mmap can sometimes cause issues - any specific reason you're using it? And presence-penalty 1.5 is pretty high, might contribute to loops. Try lowering to 1.2 or 1.1. Also I had issues with Qwen3.5 and mmproj before, have you tried running it without this flag? Let me know if that helps!

[-]

PsychologicalSock239@reddit

what does "--swa-full" and "--ctx-checkpoints 512" do?

[-]

thejacer@reddit (OP)

It seems to be doing well with —jinja added. I didn’t do a complete rebuild cause I wanted to know if the current version acted better with jinja. It looks like that did it. Thanks. I haven’t been using jinja cause I thought it was enabled by default

[-]

ShuraWW@reddit

No worries, glad to hear it worked!

[-]

thejacer@reddit (OP)

Oh also, I added no-mmap a while back because I couldn’t get any models to load without it. Since I moved to docker I need to try without it

[-]

El_90@reddit

In last 48 hours changes the new UDQ5XL wont load (same params) so dropped to Q5K, and that is taking several hours of processing to do anything, I might be in the same boat

[-]

yoracale@reddit

Are you sure it's not because your context length is too short? The model is now 2GB larger

[-]

thejacer@reddit (OP)

I’m not explicitly setting my context though

[-]

yoracale@reddit

UD_K_XL for 35b didn't get updated today though...the last update was 6 days ago. Are you sure it's not just a coincidence or placebo OP u/thejacer?

[-]

JamesEvoAI@reddit

What version of llama.cpp? What GPU? What exact sampling settings?

[-]

thejacer@reddit (OP)

I built the docker three days ago. Not entirely sure what version exactly but it was the newest then. I’ll check when I get home. 2x Mi50. Temp 1.0, presence 1.5…can’t remember the rest. It’s straight from the model card.

[-]

thejacer@reddit (OP)

I'm on release 8189, here are my parameters: \-m /models/Qwen3.5-35B-A3B-UD-Q6\_K\_XL.gguf \--host [0.0.0.0](http://0.0.0.0) \--port 8080 \-ngl 99 \--mmproj /models/mmproj\_Qwen3\_5-F16.gguf \--no-mmap \-fa on \--api-key key \--temp 1.0 \--top-p 0.95 \--top-k 20 \--min-p 0.0 \--presence-penalty 1.5 \--repeat-penalty 1.0

[-]

coder543@reddit

Huggingface has full commit history. You can download any version of any GGUF that you want, not just the latest ones.

[-]

thejacer@reddit (OP)

Oh shi…that’s true! Oh man thank you!