Jackrong/Qwopus3.5-9B-Coder-GGUF · Hugging Face

Posted by pmttyji@reddit | LocalLLaMA | View on Reddit | 72 comments

>**Qwopus3.5-9B-coder** is specially optimized and fine-tuned for high-performance **🤖 Agentic Coding, complex Tool Calling, and logical reasoning.** >*💡* ***Why the 9B Dense Model?*** *We believe that the 9B dense architecture represents the perfect* ***"sweet spot"*** *for large language models. It runs seamlessly at 8-bit precision on entry-level 16GB RAM devices—such as standard laptops and the Mac mini—making it exceptionally lightweight yet highly versatile. Without requiring expensive hardware, it allows you to achieve excellent performance paired with impressive inference speeds. Simply put,* ***Qwen3.5-9B is currently the best open-source model in its class.*** # 🛠 Training Strategy The fine-tuning process of this model deeply integrates **Trace Inversion** data augmentation technology with high-quality **Agent Traces**. This systematic approach not only strengthens the model's ability to solve complex programming tasks, but also greatly improves its logical coherence and accuracy when using various tools. This model is designed specifically for the following goals: * 🧩 More structured and stronger logical reasoning capabilities, reducing repetitive thinking * 💻 More powerful capabilities in code writing, debugging, and repository-level task processing * 🛠 More stable and accurate Tool Calling capabilities for terminal commands, file operations, and browsers * 🔁 Better cross-data source distillation alignment Check model card for all benchmarks. With MTP, hope this could be better & faster on \~10GB VRAM. Nice to do Agentic coding while getting good t/s just with 8GB VRAM.

Reply to Post

72 Comments

[-]

jopereira@reddit

How to disable reasoning in this model in llama-server? It is not responding to --chat-template-kwargs '{"enable\_thinking":false}', --reasoning off or --reasoning-budget 0.

[-]

pmttyji@reddit (OP)

Possibly these could fix [https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates) [https://huggingface.co/models?other=chat-template](https://huggingface.co/models?other=chat-template)

[-]

wingwing124@reddit

Hi all, i have been trying to use the q8 version of /u/noctrex 's model. Its pretty amazing, except I am getting that classic qwen3.5 looping behavior all over the place. I figured this wouldn't be so bad on q8 but it is. Can anyone point me to some of the most likely culprits for this behavior that I can tweak? I'd really like to use this model

[-]

noctrex@reddit

On what agent are you seeing this behavior? Also, can you try out this template, to see if fixes it? [https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates) I'm using this template and I've tested sessions with 128k Q4 KV cache, filling it up to 90k, and it seems to go smoothly without looping.

[-]

wingwing124@reddit

I tried this template, but am unfortunately still getting agentic loops. Particularly when a tool call or a write fails, for whatever reason. Other times it will loop over a series of tasks (Read file 1, read file 2, read file 3, read file 1, read file 2... And so on). I am pretty sure this is just Qwen3.5 being Qwen3.5, but I'm really hoping I can get it going soon! Your version is crazy fast

[-]

noctrex@reddit

Well, that's unfortunate. Where are you using this model? With what back end? Do you have any objections to sharing your command?

[-]

wingwing124@reddit

I will try this out later today, thank you! I am seeing this on both pi and hermes. Running 64k context, unquantized kv cache

[-]

wowsers7@reddit

Have you tested it with SmallCode? Might be a potent combo. [https://www.reddit.com/r/LocalLLM/comments/1tged8r/i\_built\_a\_coding\_agent\_that\_gets\_87\_on\_benchmarks/](https://www.reddit.com/r/LocalLLM/comments/1tged8r/i_built_a_coding_agent_that_gets_87_on_benchmarks/)

[-]

pmttyji@reddit (OP)

Not yet. Haven't tried agentic coding before as my current laptop is not strong enough(just 8GB VRAM). How's it for you?

[-]

plasm0r@reddit

On my 5070 Ti 16GB VRAM, Ryzen 7600, 32GB System RAM I get: For Qwen3.5-9B: 82 tps For Qwopus3.5-9B-Coder-MTP: 153 tps Config used: `llama-server ^` `--model "C:\models\Qwopus3.5-9B-Coder-MTP-Q8_0.gguf" ^` `--host` [`0.0.0.0`](http://0.0.0.0) `^` `--port 8080 ^` `--ctx-size 131072 ^` `--n-gpu-layers all ^` `--fit on ^` `--batch-size 512 ^` `--ubatch-size 128 ^` `--threads 6 ^` `--threads-batch 12 ^` `--parallel 1 ^` `--cont-batching ^` `--metrics ^` `--jinja ^` `--spec-type draft-mtp ^` `--spec-draft-n-max 3 ^` `--temp 0.3 ^` `--top-p 0.95 ^` `--top-k 20` Very nice performance increase :)

[-]

soyalemujica@reddit

I wonder why in this model range and not in 3.6 27B ?

[-]

noctrex@reddit

It exists: [https://huggingface.co/Jackrong/models?search=qwopus3.6](https://huggingface.co/Jackrong/models?search=qwopus3.6) I've made MTP versions: [https://huggingface.co/noctrex/models?search=qwopus3.6](https://huggingface.co/noctrex/models?search=qwopus3.6)

[-]

soyalemujica@reddit

Is you the noctrex ? I didn't know you used a 7900XTX in general, that means that every NVFP4 quantification model you have you built is for this card in general ?

[-]

noctrex@reddit

>Is you the noctrex ? yup its me ! >I didn't know you used a 7900XTX in general It has served me very well. It's essentially my gaming rig. >that means that every NVFP4 quantification model you have you built is for this card in general ? Those are ***MX***FP4 quants, not ***NV***FP4, as those are only compatible with nvidia cards. The MX one its the open standard that is compatible with all machines. >I see that there's only a 4QS model, no plans for a Q5KM or Q4KM? Yes, because testing it, it has served me very well. it seems to be better than Q4KM, and Q5KM does OOM, with any useful context size (>64k). Do you want me to quant one for you to test it?

[-]

thoquz@reddit

Thank you for making these brilliant MTPs! So for your MTP 27B-V1-preview, are you running it at IQ4? How much context are you still able to have with MTP on? I'd love to see your 7900 XTX 24GB settings if you're willing to share.

[-]

noctrex@reddit

I'm using the following configuration successfully with pi agent, in my llama-swap config: C:/Programs/AI/llamacpp-vulkan/llama-server.exe --port 8080 --flash-attn on --jinja --ctx-checkpoints 256 --parallel 1 --model Q:/Models/LLM/Qwopus3.6-27B-v1-preview-MTP-IQ4_XS.gguf --cache-type-k q4_0 --cache-type-v q4_0 --ctx-size 131072 --fit-ctx 131072 --temp 0.6 --top-p 0.95 --top-k 20 --min_p 0.00 --presence-penalty 0.0 --repeat-penalty 1.0 --chat-template-kwargs "{\"preserve_thinking\": true}" --spec-type draft-mtp --spec-draft-p-min 0.75 --spec-draft-n-max 3 --chat-template-file C:/Programs/AI/llama-swap/templates/Qwen3.6-chat_template-v19.jinja I'm also using the chat template from [https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates)

[-]

thoquz@reddit

Brilliant, thank you! Are you happy with the performance of the Q4 cache for coding? I suspect it might recall a long function wrong sometimes. I've managed to do a similar context size (also 131k) with Q8 and no MTP on the unsloth Quant's iq4_xxs. (No vision layer) Do you know much vram or context the MTP layer takes?

[-]

noctrex@reddit

It actually has been surprisingly very stable with Q4. I have already done multiple sessions where the context fills to 90k+ and it seems to remember. Actually, I have been very surprised because gemma gets very stupid very quick, and it's unusable for me, whereas they cooked a masterpiece with qwen3.6

[-]

thoquz@reddit

Wow! I guess it's the magic of Qwen 3.6's gated delta net attention, compared to the Gemma's sliding window attention.

[-]

noctrex@reddit

That, and also the new KV cache rotation that ggerganov implemented in llama.cpp, that makes Q4 KV useable [https://github.com/ggml-org/llama.cpp/pull/21038](https://github.com/ggml-org/llama.cpp/pull/21038)

[-]

grunade47@reddit

Hi noctrex, im having trouble using the latest version of llama.cpp with LM studio and the model (with MTP) is failing to load too. Im gonna try now with the non-MTP version Can you help me setup both please

[-]

noctrex@reddit

Unfortunately, I don't really have any experience with LM Studio. I use llama.cpp exclusively with llama-swap.

[-]

grunade47@reddit

ah okay, no worries i'll try installing it directly too

[-]

soyalemujica@reddit

Q4KM would not hurt at all :D

[-]

soyalemujica@reddit

With the 7900XTX, IQ4S that noctrex uploaded you can fit 128k with Q8 kvcache, with the Q5KM model you can fit 128k with Q4 kvcache

[-]

thoquz@reddit

Wow, even better than I expected? Which option do you personally prefer out of those two? I expected originally the MTP layers would eat a few GB of VRAM

[-]

noctrex@reddit

Yes, you can also fit it with Q8, but it gets very close to the limit, but as I'm using the card also for my display, it tends to OOM if I have open some windows. If you have a dedicated 24GB card for inference you can fit it.

[-]

soyalemujica@reddit

I have to report, using the 4QS model, I unfortunately experienced the a thinking loop during file scan, like "let me read file..." it would loop there endlessly until I cancelled it, no idea if this is due to the 4QS quant but I have never experienced this behavior before with the 27B model

[-]

noctrex@reddit

I should also mention that I am using this chat template in my llama-swap config: [https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates) Where did you see the looping? I'm using pi agent, and it hasn't happened for me yet. I'll cook a quant in a minute.

[-]

soyalemujica@reddit

Also using Pi Agent, it got stuck trying to read a file, it kept repeating the same file read over and over again with the same message. Ima give a try to this chat template. However, I was also using ngram-mod + mtp-draft, I am now only using mtp-draft and it seems to be fine. Thank you for these quantifications! Also appreciate it that you're cooking a higher quant model, I need that for C++ precision

[-]

soyalemujica@reddit

I have been running Q5KM with 128k context at Q4\_0 KVCache just fine for C++, literally have not experienced any hallucination, and I'm dealing with game coordinates x/y/z and a lot of C++ without any OOM issues (ofc using Linux), tbh I would not mind a Q5KM model in general with MTP under your bare hands. Yeah, I meant MXFP4, I'm sorry.

[-]

pmttyji@reddit (OP)

Jackrong might release other (coder) variants soon or later possibly as this one came out just 2 days ago. Lets see

[-]

Hypilein@reddit

Can you actually do quality coding with a 9b model? And if yes, under which circumstances? I couldn’t even get qwen 3.5 9b to mark a grade 6 geography exam properly.

[-]

Randomdotmath@reddit

First off, I don’t really get why you’d use a geography exam to judge a 9B model. That’s more about knowledge than intelligence. Just hook it up with Wikipedia. Also, yes, a 9B model can write *any* code very well once you give it proper linting, formatting, testing tools, and framework-specific skills. Honestly, you shouldn’t trust even a 1T model to produce high-quality code without those things.

[-]

Hypilein@reddit

Let’s just say the problem wasn’t lack of knowledge.

[-]

noctrex@reddit

I have added MTP to it: [https://huggingface.co/noctrex/Qwopus3.5-9B-Coder-MTP](https://huggingface.co/noctrex/Qwopus3.5-9B-Coder-MTP) Seems to be very fast.

[-]

New_Comfortable7240@reddit

For my 3060 12 GB VRAM this model worked better than the 35B one using MTP! At least in tps. Look this summary from the AI: | Metric | Dense (Qwen 3.5-9B MTP) | MOE (Qwen 3.6-35B-A3B) | |--------|------------------------|------------------------| | **PP (first request)** | 1420.95 tokens/s | 597.40 tokens/s | | **PP (cached requests)** | 708-1284 tokens/s | 427-501 tokens/s | | **TG (generation)** | 31.79-37.67 tokens/s | 25.77-27.75 tokens/s | | **Speculative Decoding** | ✅ MTP (73-89% acceptance) | ❌ None | | **Context Checkpoints** | 32 max | 64 max | | **Checkpoint Size** | ~76-112 MiB | ~62.8 MiB | ### Key Differences **Prompt Processing (PP) - Dense is ~2.4x faster** - **Dense first run**: 1420.95 tokens/s (12602 tokens) - **MOE first run**: 597.40 tokens/s (12992 tokens) - The dense model's advantage comes from having all parameters in a single expert path vs. the routing overhead in MOE **Token Generation (TG) - Dense is ~28% faster** - **Dense**: 31.79-37.67 tokens/s (average ~34 t/s) - **MOE**: 25.77-27.75 tokens/s (average ~27 t/s) The dense model benefits from MTP speculative decoding: ``` draft acceptance rate = 0.73485 (first run) 0.89000 (second run) 0.87037 (third run) ``` This effectively adds ~15-20 t/s of "free" tokens via speculative decoding. **MTP Impact** The MTP implementation provides significant acceleration: - First request: 97 accepted / 132 generated → **73.5% acceptance** - Later requests: 89% acceptance - Final requests: 51-70% acceptance (longer outputs tend to have lower acceptance) The draft generation time is negligible (~0.2-1.6ms) vs. main model evaluation (~6.5-15 seconds). The MTP speculative decoding is highly effective here, turning a ~34 t/s base speed into effectively ~50+ t/s when you count the accepted drafts. The MOE model has no speculative decoding enabled, which is a significant missed opportunity.

[-]

New_Comfortable7240@reddit

One detail, I notice 35B is more "intelligent", but for agentic use the 9B worked fine, maybe for planning and debugging I will contiounue using 35B and for agentic use or quick stuff 9B

[-]

pmttyji@reddit (OP)

Hey! Thanks for those MTP GGUFs. Could you please make one more MTP GGUF for [https://huggingface.co/Tesslate/OmniCoder-9B](https://huggingface.co/Tesslate/OmniCoder-9B) ? One more 9B Coder

[-]

noctrex@reddit

I don't see the value of doing this one. They said some time ago that they will release the 2nd version of it, but no news of it yet. I find Qwopus3.5-9B-Coder more capable anyway.

[-]

pmttyji@reddit (OP)

>I don't see the value of doing this one. They said some time ago that they will release the 2nd version of it, but no news of it yet. Yeah, they removed v2 version quickly after release. I remember that too. Probably they're waiting for 3.6 version. Thanks

[-]

Valuable_Touch5670@reddit

If I am not mistaken, your Q8 version does NOT use the imatrix, right?

[-]

noctrex@reddit

Yes, the Q8\_0 quantization in general does not make use of an imatrix, it's not a K quant. The largest quant that does use imatrix is Q6\_K.

[-]

Valuable_Touch5670@reddit

Thanks! Will give the Q8 a shot.

[-]

bobaburger@reddit

Looking forward for a 27B version of this! I know it takes a lot to fine tune these large models but hopefully it will become a reality soon. Haven’t try this version yet but in general, small models like this tends to be bad at holding context while working, for example, it would check and see the project implemented in Rust, but will still trying to read files as Js or python in the tool calls. And many other issues similar to this.

[-]

Christosconst@reddit

Hey u/pmttyji, llama-cli works fine with the model, but if I run it with llama-server, it goes into a strange infinite loop that seems it might be caused by internal instructions. See [https://gist.github.com/snez/d284df1867f371f33d83049ed9990a10](https://gist.github.com/snez/d284df1867f371f33d83049ed9990a10) for an example

[-]

exodusTay@reddit

Stuff like this makes me very hyped for the next two years. Where we are going we won't need mega cloud providers.

[-]

sqquima@reddit

We'll just need 1.21 jigowatts, or Mr. Fusion.

[-]

exodusTay@reddit

Not without 5 petabytes of VRAM with 1TB/s memory bandwith.

[-]

Amazing_Athlete_2265@reddit

Curious to see how this model stacks up against omnicoder 9b and th qwen3.6 35B MoE. Might have a play around with pi coding harness when time allows. Thanks for sharing!

[-]

ea_man@reddit

Well as a dense model it's supposed to take better use of MTP, as for quality / speed that depends: are you willing to load an A3B IQ3 all in 16GB vram?

[-]

OsmanthusBloom@reddit

Is there any reason to use this instead of Qwen3.6-35B-A3B with partial CPU offload? (assuming you have the RAM for that)

[-]

letsgoiowa@reddit

You can't use that big of a model on a 16 GB device.

[-]

pmttyji@reddit (OP)

This 9B(Q4) is good for low VRAMs like 4GB or 6GB so it fits VRAM. 35B is too big for those low VRAMs. Even too big for 8GB VRAM(Q4 is 17GB).

[-]

OsmanthusBloom@reddit

I'm running 35B-A3B on a 6GB VRAM laptop, partially offloaded to RAM. Just posted some MTP benchmark results: [https://www.reddit.com/r/LocalLLaMA/comments/1tfq683/mtp\_for\_qwen3635ba3b\_on\_6gb\_vram\_laptop\_not\_worth/](https://www.reddit.com/r/LocalLLaMA/comments/1tfq683/mtp_for_qwen3635ba3b_on_6gb_vram_laptop_not_worth/) But yes, I take your point. 9B model fully in VRAM is probably faster than 35B partially offloaded.

[-]

pmttyji@reddit (OP)

Also MTP gives more boost for Dense models(than MOE models)

[-]

OsmanthusBloom@reddit

Good point!

[-]

icedgz@reddit

Faster but is it worth it to take the hit for the smarter model

[-]

Hot-Employ-3399@reddit

For one vulkan llama inference is shit with offload. I from the start get less than 10 tokens per second on 5090 🤡 (And I love me some vulkan because it causes significantly less temperature raise on my laptop with the same token generation speed when no offload is used)

[-]

Constant-Simple-1234@reddit

Does it have MTP in the model? I thought you need to convert it or sth. As when I load the regular ones it gives error.

[-]

Beginning_Ad_3309@reddit

I have been on vocation for 2 weeks, and now I don't know What is MTP?

[-]

pmttyji@reddit (OP)

[https://github.com/ggml-org/llama.cpp/pull/22673](https://github.com/ggml-org/llama.cpp/pull/22673)

[-]

ComplexType568@reddit

MTP stands for "Multi-Token Prediction", it's meant to be a lossless form of Speculative Decoding where the main model will quickly guess the next few tokens and self-validate (from what I know), it requires no new "draft model" (unlike traditional speculative decoding), it is usually most effective in dense models rather than MoE models and speeds vary from domain to domain.

[-]

Alex_L1nk@reddit

I've downloaded this one and seems to work fine: [https://huggingface.co/noctrex/Qwopus3.5-9B-Coder-MTP](https://huggingface.co/noctrex/Qwopus3.5-9B-Coder-MTP)

[-]

pmttyji@reddit (OP)

Nope. I think these GGUFs have been created 1 day before MTP merge on llama.cpp. Maybe [post a message there](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder-GGUF/discussions) for MTP.

[-]

pedronasser_@reddit

I can't wait for Qwopus3.6 35B A3B MTP. Qwopus is already faster/smarter than the base model. Imagine with MTP.

[-]

pmttyji@reddit (OP)

u/noctrex already created 2 GGUFs, check it out [https://huggingface.co/noctrex/models?sort=created&search=mtp](https://huggingface.co/noctrex/models?sort=created&search=mtp)

[-]

ResidentPositive4122@reddit

> It runs seamlessly at 8-bit precision on entry-level 16GB RAM > With MTP, hope this could be better & faster on ~10GB VRAM. > Nice to do Agentic coding while getting good t/s just with 8GB VRAM. This feels like that joke with the cops finding 10lb of drugs. It'd be a shame if those 8lb found their way onto the streets, so they ended up burning 6lb of drugs to get rid of them.

[-]

pmttyji@reddit (OP)

My bad :D , That 8GB is what I have(so I'll go with Q6). 10GB is good to fit Q8 on VRAM.

[-]