Qwen 3 Coder Next has a bug! Help Test?
Posted by CSEliot@reddit | LocalLLaMA | View on Reddit | 38 comments
Hey y'all.
So I've stumbled upon a really specific and esoteric "bug" where an llm can't comprehend a URL in like, 90% of scenarios.
Processing img qqe0qn928dvg1...
I can easily get OTHER llms to accurately play with this URL. But I've tried qwen, unsloth, and bartowski quant of Qwen 3 Coder Next and they ALL struggle with this url. I can't even get the ai to do a 'visit website' on this url.
Notice in the screenshot the added "d" in "rocm" after the https://.
Here's the url: https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html
It also tends to lose the /en/.
If anyone else can test this ... if I can get at least one other person to verify this for me, I'll submit an issue on their github https://github.com/QwenLM/Qwen3-Coder/issues .
Thanks!
fragment_me@reddit
Interesting. Did you try the CLI vs web UI?
CSEliot@reddit (OP)
Qwen 3 Coder Next isn't available on their web ui.
fragment_me@reddit
Get off that trash UI
DataGOGO@reddit
Works just fine on my machine: Qwen3 Coder Next FP8 / NVFP4.
CSEliot@reddit (OP)
Do you have a hugging face link I can test it out?
DataGOGO@reddit
GadflyII/Qwen3-Coder-Next-NVFP4 · Hugging Face
I never published the FP8 weights.
CSEliot@reddit (OP)
Ah, I can't run that on my amd system sadly.
Can you double check that it works? I mean, are there are typos in the url when it replies back to you? That's what I'm talking about and perhaps I didn't make it clear.
Step 1: Pass correct URL: https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html
Step 2: Verify if AI generates back to you the SAME url WITHOUT typos like:
https://rocmdocs.amd.com/latest/compatibility/compatibility-matrix.html
See the difference between the 2?
https://rocmdocs.amd.com/latest/compatibility/compatibility-matrix.html
https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html
DataGOGO@reddit
there is no difference. I tried the original model, FP8 and NVFP4 quants, all behave exactly the same.
With all three, it parses the URL, performs the tool call, and returns the results with the original URL set as the source.
no URL changes.
CSEliot@reddit (OP)
Hmmmm, perhaps the bug is more related to the llama.cpp vulkan build I'm using. Thank you for testing!
DataGOGO@reddit
yeah, you found a strange one, I highly recommend you try with vLLM or SGLang as a next step; my knee jerk is that it is llama.cpp.
qubridInc@reddit
Sounds like a tokenization/URL parsing issue worth reporting, especially if it consistently mangles structured links.
CSEliot@reddit (OP)
It works with a lot of different quants, but according to what others have shared it's perhaps more related to the version of vulkan-llama.cpp that I'm using. The llama.cpp runtimes that LM Studio provides tend to be a couple weeks behind, so before I report it i'd have to be able to recreate it in the latest llama.cpp that i build myself. ... which if I'm being honest I probably dont have time for.
ThisNameWasUnused@reddit
This is not a Qwen 3 Coder issue; it's the UI you're using issue. It's not able to use any tools to connect to webpages because maybe the UI you're using has no capability to do so, thus, it giving you information based on its training data. It's not hallucinating.
An app that I created to chat with LLMs using llama.cpp backend successfully summarized that URL with Coder Next that I loaded.
CSEliot@reddit (OP)
I got an unsloth quant that didn't produce the issue, so I'm gonna stick with that unless I ever feel like testing again.
CSEliot@reddit (OP)
Which version of Coder Next did you use? I'll look it up on hugging face and give it a shot.
ThisNameWasUnused@reddit
The one from Unsloth.
CSEliot@reddit (OP)
Sorry, which Quant? Downloading their UD Q4_K_XL right now.
ThisNameWasUnused@reddit
I'm using UD Q8_K_XL, but my system has 96 GB VRAM.
CSEliot@reddit (OP)
Can you double check that it works? I mean, are there are typos in the url when it replies back to you? That's what I'm talking about and perhaps I didn't make it clear.
Step 1: Pass correct URL: https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html
Step 2: Verify if AI generates back to you the SAME url WITHOUT typos like:
https://rocmdocs.amd.com/latest/compatibility/compatibility-matrix.html
See the difference between the 2?
https://rocmdocs.amd.com/latest/compatibility/compatibility-matrix.html
https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html
ThisNameWasUnused@reddit
You can see that it works in the screenshot.
As for the URL difference, ` https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html ` and ` https://rocmdocs.amd.com/en/latest/compatibility/compatibility-matrix.html ` point to the same page. AMD, for whatever reason, has both versions of the main URL. The model I used shows the URL that you're referring to, but in your case, it's a dead link because it's missing the "/en/".
CSEliot@reddit (OP)
Thank you for your time! If I could give you 2 upvotes I would!
So, it turns out that my llms were all hallucinating an incorrect url, DESPITE what you were pointing out. Yours modified the url to the following:
https://rocmdocs.amd.com (as you stated)
Mine modified to the following:
https://rocmd.docs.amd.com
Aaaaaaand an unsloth quant is not producing these hallucinations. This is one of those "frustrating it works but dont know why" situations ...
Guess I'm sticking with unsloth again! I swapped to Bartowski after having some repetition and tool issues and wanted to try another quant group.
again, thanks for your time!
yoracale@reddit
Yep you're right, it worked everytime when i tried it so it's definitely tooling related
CSEliot@reddit (OP)
It's not working though as it's literally typo'd in your screenshot, exactly as I've explained in my original post. WHY CAN NOBODY READ. Humanity is doomed.
yoracale@reddit
It's not a typo, amd has two links that work: https://rocmdocs.amd.com/en/latest/compatibility/compatibility-matrix.html
And https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html
They both have the same content
CSEliot@reddit (OP)
HAH. That's crazy. Okay, you're right. My mistake.
Your version of Coder Next produced a url that was correct.
Mine though is not sadly: https://rocmd.docs.amd.com/en/latest/compatibility/compatibility-matrix.html
\^ it's a subtle difference, rocmdocs (yours) vs rocmd.docs (mine).
CSEliot@reddit (OP)
If you look at u/yoracale s screenshot, you can see it in fact broke for them as well. The url that the ai spits out is NOT the same as the one passed. That url 404s.
CSEliot@reddit (OP)
This is LM Studio. I use it w web search and website visiting tools daily.
Odd-Ordinary-5922@reddit
Does it actually have a tool to search?
CSEliot@reddit (OP)
Tool or no tool it hallucinated.
ComplexType568@reddit
That is not a bug that is a fundamental behavior the model has
CSEliot@reddit (OP)
To hallucinate? What?
Ell2509@reddit
Im a new contributor but yes. Hallucination is a feature of LLMs, not a bug. They navigate meaning entirely in abstract. Even their correct answers are hallucinations, technically. But no, hallucinating is not a bug. It is 100% a feature. Now you are in to controlling LLM behaviour, which is what rhe frontier labs are wrestling with too.
CSEliot@reddit (OP)
Yes no ai is actually intelligent. It's all a statistical likelihood of next-token. But when we say "hallucinate" we mean "said something very incorrect".
Odd-Ordinary-5922@reddit
so you dont have a tool yet you are asking it to search the url?
CSEliot@reddit (OP)
I've tested the various quants but with tool usage and without tool usage and they all hallucinate when generating the required url. I can share more screenshots if you'd like.
yoracale@reddit
I just tried it in unsloth studio, works for me
CSEliot@reddit (OP)
Edit: WAIT NO IT DIDNT WORK FOR YOU EITHER.
Ironically, I'm in the middle of struggling to get unsloth studio working. I'm on an AMD Strix Halo machine. Will report back if i can get it to work. Thanks for testing!
CSEliot@reddit (OP)
Ironically, I'm in the middle of struggling to get unsloth studio working. I'm on an AMD Strix Halo machine. Will report back if i can get it to work. Thanks for testing!