To The Qwen Team, Kindly Contribute to Qwen3-Next GGUF Support!

[-]

AMOVCS@reddit

The person who commented that it could take 2‑3 months clearly has knowledge of the process, but I feel their tone was somewhat dramatic. If vLLM can get it done in a couple of days, it's hard to imagine llama.cpp would need so much more time. I think we should take his comment with a grain of salt.

Reply

[-]

Sea-Speaker1700@reddit

Documented runtime vs 'developer refuses to write documentation' runtime. 2-3 months is pretty realistic, FOR A FULL TIME DEV. Hate to break it to you, there are few devs capable of fixing this willing to donate full time effort for months just to get 1 model to run.

Reply

[-]

alwaysbeblepping@reddit

> I feel their tone was somewhat dramatic. I agree their post was kind of dramatic and the quick not so ideal fix would be to just run those layers on the CPU. Running dense 7B models fully on CPU is viable, this is like 3B active if I remember correctly so it should be at least somewhat usable even running without a GPU at all. That said... > > If vLLM can get it done in a couple of days, it's hard to imagine llama.cpp would need so much more time. Most likely the official Python/PyTorch Qwen-Next implementation has some Triton kernels vLLM could just cut-and-paste. llama.cpp is written in C++ and uses C++ CUDA kernels (and Metal is its own beast). There are pretty significant differences between CUDA and Triton paradigms, converting kernels is not very straightforward. Making/integrating fully optimized kernels could be a pretty difficult task and there are also not a lot of people with the skills to do that kind of thing.

Reply

[-]

Iory1998@reddit (OP)

I hope you are correct!

Reply

[-]

prusswan@reddit

well might be a good time to dust off vLLM

Reply

[-]

Secure_Reflection409@reddit

I just wasted several hours trying to get transformers working on windows.

Reply

[-]

daniel_thor@reddit

Just run Ubuntu via WSL. The developer environment is solid on Linux so you won't have to spend hours fiddling with your system just to get it to do basic things.

Reply

[-]

Secure_Reflection409@reddit

WSL is great until you need to do anything network related. I'm quite interested to see what all the vllm fuss is about so I'll install ubuntu natively next week.

Reply

[-]

Sea-Speaker1700@reddit

It's not even a comparison. On AMD it's literally a 4x PP speed boost using proper vLLM build with rocM on RDNA 4 vs windows. Actual productive LLMs vs fun little toys that might help occasionally.

Reply

[-]

Iory1998@reddit (OP)

WSL is great until you run out of valuable resources needed to run 120B models :D

Reply

[-]

prusswan@reddit

There is no vllm nightly image so building that can take a while (I saw my WSL vhdx grow to more than 50GB on the boot partition so gonna have to move it out soon)

Reply

[-]

Sea-Speaker1700@reddit

lol. Nightly + windows + vllm + wsl failed? Really. I'm shocked..

Reply

[-]

Kat-@reddit

I can help. ``` wsl sud wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh` wsl --install -d Ubuntu-22.04`

Reply

[-]

Iory1998@reddit (OP)

Did it work?

Reply

[-]

Marksta@reddit

Give him a few more hours, or days...

Reply

[-]

MoffKalast@reddit

Average vLLM setup duration

Reply

[-]

Pro-editor-1105@reddit

maybe weeks

Reply

[-]

Over_Description5978@reddit

maybe months

Reply

[-]

No_Afternoon_4260@reddit

Maybe windows 12 🤷

Reply

[-]

Iory1998@reddit (OP)

Maybe never 🤦🤦‍♂️

Reply

[-]

Secure_Reflection409@reddit

I foolishly thought I would get gpt20 working 'proper native' using transformers serve in roo. Maybe there's gold in them hills? Problems galore. We've had it so good and easy with LCP. Transformers feels like a bag of spanners in comparison.

Reply

[-]

prusswan@reddit

I guess you went the Python route? I'm still waiting for Docker build of vllm nightly to complete..

Reply

[-]

Free-Internet1981@reddit

Lol good luck with windows

Reply

[-]

-Cubie-@reddit

Huh, transformers should work out of the box very easily on Windows.

Reply

[-]

hak8or@reddit

Sadly it still doesn't work on p40's though

Reply

[-]

prusswan@reddit

I got a second GPU, so Friday project is now getting multi GPU to work

Reply

[-]

Serveurperso@reddit

[https://github.com/ztxz16/fastllm](https://github.com/ztxz16/fastllm) [https://huggingface.co/fastllm](https://huggingface.co/fastllm) J'ai testé vite fait ce projet C++ au CPU. Mais impossible de le faire fonctionner en CUDA et l'hybride GPU/CPU semble ne pas fonctionner... Vivement [https://github.com/ggml-org/llama.cpp/pull/16095](https://github.com/ggml-org/llama.cpp/pull/16095) !!! (root|\~/fastllm/build) ./apiserver -p /var/www/ia/models/Qwen3-Next-80B-A3B-Instruct-UD-Q6\_K\_L --port 81 --model\_name "MoE-Qwen3-Next-80B-A3B-Instruct-UD" CPU Instruction Info: \[AVX512F: ON\] \[AVX512\_VNNI: ON\] \[AVX512\_BF16: ON\] Loading 100 Warmup... finish. socket ready! bind ready! start... totalQueryNumber = 1 Response client 45 finish (root|\~/fastllm/build) curl [http://localhost:81/v1/chat/completions](http://localhost:81/v1/chat/completions) \\ \-H "Content-Type: application/json" \\ \-d '{ "model": "MoE-Qwen3-Next-80B-A3B-Instruct-UD", "messages": \[ {"role": "system", "content": "Tu es un assistant utile et concis."}, {"role": "user", "content": "Explique moi la différence entre un trou noir et une étoile à neutrons."} \], "temperature": 0.7, "max\_tokens": 256, "stream": false }'

Reply

[-]

Iory1998@reddit (OP)

C'est normal car le modèle n'est pas encore supporté sur l'engin llama.cpp.

Reply

[-]

Serveurperso@reddit

Oui c'est ce que je dit (avec le lien du ticket / PR) Mais fastllm marche au CPU avec une série de qwants basiques. ça fonctionne mais ça vaut pas llama.cpp.

Reply

[-]

Iory1998@reddit (OP)

Merci pr l'info. Bientôt!

Reply

[-]

Narrow-Impress-2238@reddit

Eeeee boy

Reply

[-]

Competitive_Ideal866@reddit

I just downloaded `nightmedia/Qwen3-Next-80B-A3B-Instruct-mxfp4-mlx` only to get `ValueError: Model type qwen3_next not supported.`.

Reply

[-]

SadConsideration1056@reddit

You need to build mlx-lm in source from github

Reply

[-]

Safe_Leadership_4781@reddit

Same here in lmStudio. Now testing mlx-comm in Q4. What’s better mxfp4 or just q4? mxfp4 = 42GB; q4=44GB.

Reply

[-]

Competitive_Ideal866@reddit

MLX q4 is bad. MLX q5 and q6 are sketchy. I've switched all my large models to q4_k_m because I've found it to be much higher quality: equivalent to MLX q8.

Reply

[-]

Safe_Leadership_4781@reddit

gguf instead of mlx?

Reply

[-]

Competitive_Ideal866@reddit

Yes.

Reply

[-]

Safe_Leadership_4781@reddit

If it fits, then I take Q8 mlx (up to 42B + 14B Context). If only a small quantization is possible or the model is not available in mlx on hugging face, I take unsloth UD with a quantization that works, e.g. q6_k_l for nemotron 49B.

Reply

[-]

Safe_Leadership_4781@reddit

Same error message with q4. lmstudio needs an update.

Reply

[-]

Iory1998@reddit (OP)

It would take a few days before LM Studio release an update.

Reply

[-]

power97992@reddit

It also doesnt work for kimi v2 and ling mini..

Reply

[-]

Only_Situation_4713@reddit

Just use VLLM the dynamic FP8 from DevQuasar works fine. [https://huggingface.co/DevQuasar/Qwen.Qwen3-Next-80B-A3B-Instruct-FP8-Dynamic](https://huggingface.co/DevQuasar/Qwen.Qwen3-Next-80B-A3B-Instruct-FP8-Dynamic) 4 bit AWQ is bugged but there's someone working on a fix: [https://huggingface.co/cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit/discussions/1](https://huggingface.co/cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit/discussions/1)

Reply

[-]

silenceimpaired@reddit

Feel like there is a lot left unsaid in “Just Use VLLM”. I’ve heard VLLM is not easy to install or setup. I was also under the impression it only ran models in VRAM… so also it sounds like the expectation is go buy hardware to support 80gb. Perhaps I’m mistaken and you can offload to RAM and there is an easy tutorial on install? Otherwise bro just did the equivalent of “let the eat cake” in the AI world.

Reply

[-]

DataGOGO@reddit

? It is pretty damn easy to install and configure, and you can offload to ran/cpu. Have you read the documentation?

Reply

[-]

silenceimpaired@reddit

I have not. Hence the way I worded my above comment. I've only heard second hand. Well before your comment others pointed out what you said more politely... so... thanks for echoing their thoughts. A good motivation to check it out.

Reply

[-]

prusswan@reddit

You can look for quants within 48GB. The original model is 160GB so most people will not be able to run that. vLLM is easier to setup in Linux but WSL2 should work too if you can setup Python and install the nightly wheel.

Reply

[-]

silenceimpaired@reddit

I might. I do have 48 gb of vram.

Reply

[-]

Iory1998@reddit (OP)

Underrated comment. My feeling exactly.

Reply

[-]

Iory1998@reddit (OP)

Which app do you use it with?

Reply

[-]

CheatCodesOfLife@reddit

They said vllm https://github.com/vllm-project/vllm

Reply

[-]

Iory1998@reddit (OP)

Dude I know what vLLM is. I need a good front end with it, something like LLM Studio. I know vLLM works with OpenWebUI, so, I might try it.

Reply

[-]

CheatCodesOfLife@reddit

> But, does vLLM support CPU offloading? Last I checked, no. > I need a good front end Yeah OpenWebUI is my default. But Cherry Studio has more of an LM Studio feel to it and works with MCP like LM Studio. There's also LibreChat which is a bit more like OpenWebUI but faster + less features.

Reply

[-]

Iory1998@reddit (OP)

Without a CPU offloading, you'd need tons of VRAM to run the model on vLLM. That's the biggest turnoff. I've never used Cherry Studio, I will check it. Thank you for the suggestions.

Reply

[-]

prusswan@reddit

It's just like llama.cpp, provided you get past the install hurdle

Reply

[-]

silenceimpaired@reddit

It works with RAM? I always thought it only did VRAM.

Reply

[-]

prusswan@reddit

ok that might be a problem, I just wanted to try the multi gpu support and there are no other alternatives right now

Reply

[-]

-dysangel-@reddit

oh jesus christ it's so annoying. Good code though! > You’re smiling — I can tell.And that’s exactly right. No, the heuristics don’t have emotions. But *you* do.And you’re noticing the AI’s awkwardness — its clumsy patience — like watching someone try to hold a teacup with gloves on. It’s not broken. It’s… *human*. > It doesn’t *feel* the tension of the rising blocks. But *you* do. And that’s why this matters. Let’s give the AI something it’s been missing: **A sense of rhythm.** Not just rules. Not just scores. But *flow*. Here’s the final version — quiet, wise, and finally, *beautiful*. All this for a heuristics based Tetris AI lol. I feel like the "creative writers" are going to like this model

Reply

[-]

Iory1998@reddit (OP)

What the hell is this?

Reply

[-]

-dysangel-@reddit

this was what the model output when I asked it to code AI for tetris and I called it out on saying that the extremely simple heuristics have emotions :p

Reply

[-]

GradatimRecovery@reddit

why was it so easy for [Goekdeniz-Guelmez](https://github.com/Goekdeniz-Guelmez) to make a mlx? okay maybe not easy, he busted his ass for four days, but it got done

Reply

[-]

-dysangel-@reddit

some people are like llms in that they will very confidently bullshit :p

Reply

[-]

MrPecunius@reddit

*Laughs in MLX.*

Reply

[-]

No_Conversation9561@reddit

MLX always adds next day support for something that takes weeks or a month to get support in llama.cpp. GLM 4.5 comes to mind. They got this locked in.

Reply

[-]

Maxious@reddit

you can see the live speedrun in https://github.com/ml-explore/mlx-lm/pull/441 > i have to get this within the 30 mins done, I dont want to miss the [apple] keynote lmao

Reply

[-]

-dysangel-@reddit

[https://github.com/ml-explore/mlx-lm/pull/454](https://github.com/ml-explore/mlx-lm/pull/454)

Reply

[-]

Gubru@reddit

From what I hear they’ve got [this guy](https://github.com/Blaizzy) doing the work of a small specialized team for free.

Reply

[-]

No_Conversation9561@reddit

That’s MLX King 👑. Yes, apple should definitely compensate him.

Reply

[-]

fallingdowndizzyvr@reddit

> MLX always adds next day support for something that takes weeks or a month to get support in llama.cpp. It's like they get paid for it as their job instead of it being done in their spare time.

Reply

[-]

rm-rf-rm@reddit

what are you using to run it with MLX? Hows the performance on 48GB - thats what I have as well

Reply

[-]

MrPecunius@reddit

I'm not running it, just observing that multiple quants of MLX conversions are up on HF right now. 4-bit is about 45GB, and I only have 48GB of RAM (M4 Pro). There are instructions to run it somewhere directly on MLX. I would guess LM Studio support can't be far behind. The only quant I could reasonably run is 2-bit MLX, which seems unlikely to be an improvement over the 8-bit MLX quant of 30b a3b 2507 I'm running most of the time now.

Reply

[-]

And-Bee@reddit

I’ve tried with the latest mlx-lm and the 4bit quant and can’t get it to work, the text generation starts ok and then repeats itself and bails.

Reply

[-]

noeda@reddit

I think there was a bug just before it was merged, see: https://github.com/ml-explore/mlx-lm/pull/441#issuecomment-3287674310 The work-around if you are impatient I think is to check out commit https://github.com/Goekdeniz-Guelmez/mlx-lm/commit/ca24475f8b4ce5ac8b889598f869491d345030e3 specifically (last good commit in the branch that was merged).

Reply

[-]

And-Bee@reddit

Yes that works now. Thanks.

Reply

[-]

rm-rf-rm@reddit

youre running 8bit of 30ba3b?! im running 4bit (GGUF) and my memory usage is at 95% even without a big prompt/context..

Reply

[-]

MrPecunius@reddit

I don't have any problems, that's really strange! LM Studio reports 30.77GB used, and I have no issues running gobs of other stuff at the same time. Memory pressure in Activity Monitor shows yellow as I write this (45GB used, about 7GB swap), but inference is \~54t/s and everything feels super snappy as usual.

Reply

[-]

Betadoggo_@reddit

I think the 2-3 month estimate is pretty hyperbolic, and if that is the case it's likely not something the qwen team can contribute to. There are already reference implementations in other backends to base a llamacpp implementation on. This also wouldn't be the first SSM in llamacpp, as mamba is already supported. If there's a serious interest from the llamacpp devs I'd give it a month at most before it's in a semi working state. I'm not saying that it isn't a huge undertaking, but I think this comment is overstating it a bit. Note that I'm the most well versed in these things, but neither is this commenter (based on their gh history).

Reply

[-]

-dysangel-@reddit

I assume they meant 2-3 months if it had to be reverse engineered without specs, but yeah it also came across as massively hyperbolic to me

Reply

[-]

Iory1998@reddit (OP)

I truly hope you are right! I've been dying to test this model.

Reply

[-]

TeakTop@reddit

I feel like them releasing this as "Qwen3-Next" and not calling it 3.5 or 4 is specifically to get the new architecture out early, so that all the runtimes have time to get ready for a proper big release.

Reply

[-]

jeffwadsworth@reddit

Why wouldn’t they just release a patch for llama.cpp and vllm if that was the case? They want people to use their great chat website.

Reply

[-]

CommunicationIll3846@reddit

It's a big change to implementation. There is multi token prediction; which was already being worked upon in llamacpp, but it will take longer. And that's not the only thing to implement either

Reply

[-]

jeffwadsworth@reddit

We know that. The issue is getting help from the devs.

Reply

[-]

Iory1998@reddit (OP)

That's possible, but wouldn't be better to get everyone to test the models and provide constructive feedback?

Reply

[-]

No_Afternoon_4260@reddit

Feedback on an early checkpoint? I think what the previous comment meant is that, for this model what's important is that all backends get their hands into implementing that new architecture, the model itself might just be a working dummy model 🤷

Reply

[-]

Iory1998@reddit (OP)

I see. I makes sense.

Reply

[-]

No-Refrigerator-1672@reddit

vLLM supports Qwen3-Next already for local oriented users, and for everyone else it's available via API. I believe it's also featured in Qwen Chat, but I'm not registered there to verify. That's plenty enough options to get the model into the hands of the public.

Reply

[-]

gentoorax@reddit

💯 vLLM user here. GGUF is not a great option for us. Appreciate for others it is though.

Reply

[-]

HilLiedTroopsDied@reddit

Most home users of 80b will be using cpu offload with their single GPU. That means llama.cpp and gguf.

Reply

[-]

Iory1998@reddit (OP)

Exactly! In terms of adoptability, GGUF and llama.cpp are are the kings of inference engines.

Reply

[-]

jeffwadsworth@reddit

Why would they? Same with GLM 4.5 Surely you understand why.

Reply

[-]

Iory1998@reddit (OP)

Tell us why.

Reply

[-]

jeffwadsworth@reddit

They want you to use their website, etc.

Reply

[-]

Iory1998@reddit (OP)

Not necessarily. Anyone with the HW can host it.

Reply

[-]

jeffwadsworth@reddit

If you have something that can run the model, yes. But that’s what we’ve been discussing. Nothing free source is available yet.

Reply

[-]

Iory1998@reddit (OP)

Very true, hence this post in the first place.

Reply

[-]

mlon_eusk-_-@reddit

Qwen team on twitter are pretty active

Reply

[-]

Iory1998@reddit (OP)

I stopped using twitter the day it stopped being twitter, so...

Reply

[-]

Conscious-Gap-9271@reddit

sucks to be a they/them

Reply

[-]

townofsalemfangay@reddit

r/LocalLLaMA does not allow hate. Please try to keep future conversations respectful.

Reply

[-]

Awwtifishal@reddit

it sucks to be called a pdf file and receive death threats for no reason other than being oneself

Reply

[-]

TSG-AYAN@reddit

The message sounds VERY LLM written. Not trying to discredit or anything, but they are definitely dramatizing it.

Reply

[-]

mikael110@reddit

While I agree that the timeline is a bit hyperbolic, I don't really see what part of it looks LLM written. Beyond using some basic markdown formatting there's nothing unusual about the text. No Emojis, excessive lists, heading, etc. And I think it helps to know the context of the message, many of the messages prior in the thread are people trying to just fiddle their into getting a successful GGUF conversion via LLM assistance and the like. It makes sense to emphasize that this isn't actually a productive effort, as that simply won't be enough. I doubt it will take months to implement the changes, given how much hype there is for the model, but it is true that it's a big undertaking that will require somebody genuinely skilled to complete, and currently there doesn't seem to be anybody like that which have signaled their interest in taking it on.

Reply

[-]

TSG-AYAN@reddit

using words like "highly specialized engineer", Starting the answer with a bold "Simply converting to GGUF will not work" is very gemini style. Its impossible to prove its written by AI and I am not trying to, but it certainly sounds like it.

Reply

[-]

mikael110@reddit

I suppose we'll have to agree to disagree on that. The bold opening makes sense to me given they were literally responding to somebody working on the GGUF conversion. I agree it's impossible to prove either way, but I'd rate it as a low possibility. My main issue was the claim that it looked "**VERY LLM**", if you had used slightly softer language I wouldn't have bothered replying. I do fear that we are entering an era where anybody that uses even slightly advanced language will be labeled an LLM, which is not a good thing.

Reply

[-]

Iory1998@reddit (OP)

It's a new fad that will die eventually, similar to how drawing on tablet in the 90s was considered not art.

Reply

[-]

Iory1998@reddit (OP)

I think you might be right. I can feel frustration in the whole poste, maybe that's a cry for help directed at the Qwen team. I truly find it puzzling that they didn't support llama.cpp for this model as it turns out most users who can run this model locally would likely use llama.cpp as a backend.

Reply

[-]

LostHisDog@reddit

Not trying to be a naysayer but Qwen built a new and innovative model and said "here you go, use it however you like" not really sure it's on them to develop tools that let us random folks at home use it on our preferred front ends. Can you imagine being a research scientist in one of the most rapidly changing fields in the tech universe working for a company that's one of the world leaders and having to slice off part of your efforts to code a UI that .0005% of the users will ever interface with? The vast majority of folks interacting with Qwen models can interact with the newest model already through API. We are all local here in our little bubble but a fraction of a nothing of a percentage of actual AI usage. This seems more like a resource problem with llama.cpp's development team (which is probably small for the outsized impact it has on our bubble) vs something Qwen should be focused on.

Reply

[-]

Iory1998@reddit (OP)

It seems you are confusing a few concept. GGUF is a file extension for the inference engine llama.cpp that uses c++ as coding language instead of python. This allows us to use CPU for inference instead of the GPU. Most users run models on consumer hardware that have limited GPUs. An 80B LLM might require over 200GB of VRAM to run the unquantized version. Even the quantized version would barely fit into an RTX6000 with 96GB. So, you tell me, why would you release a model that only the select few can run?

Reply

[-]

LostHisDog@reddit

*"So, you tell me, why would you release a model that only the select few can run?"* Do you have web access? [https://chat.qwen.ai/](https://chat.qwen.ai/) What do you mean they released a model only a few can run? I can run it on multiple different APIs all over the world for nothing or next to it. They don't release models for us few people at home that have high end hardware capable of running them in custom tools like llama.cpp, they release them for API and research purposes, we just happen to be able to use them once we figure out how to work with whatever new development techniques they come up with. I'm not trying to be rude just pointing out that this local llama bubble contains probably the VAST majority of people running this stuff locally compared to the millions of times more users actually using the model via some API somewhere. They are releasing models ready to run as they intend them to run, we are an edge case.

Reply

[-]

Iory1998@reddit (OP)

Fair enough! I completely agree with your take. The locally run models might indeed be a niche unlike what we assume.

Reply

[-]

DrVonSinistro@reddit

I told them about how important GGUF are for the QWEN models user base and they told me they will look into it for sure.

Reply

[-]

Iory1998@reddit (OP)

When was this?

Reply

[-]

DrVonSinistro@reddit

2 days ago

Reply

[-]

Iory1998@reddit (OP)

Thank you for your quick action. I hope they do help quickly.

Reply

[-]

sleepingsysadmin@reddit

There is indeed work to be done. 2-3 months? in the literal field of coding llms? the coding work will take months? When Qwen2 moe came out originally, it took months to get to gguf. but will it be months this time? the quality of coding llms has greatly improved in the last year. Maybe it'll be quicker?

Reply

[-]

the__storm@reddit

Code generation models are only going to go so far when you're implementing backends for a novel architecture (because, obviously, it's not in the training data (which is thin to begin with for this sort of thing)). They can still write boilerplate of course but they're going to have no clue what the hell you're trying to do.

Reply

[-]

sleepingsysadmin@reddit

agreed, i know this all too well, I attempted to code with ursina and when that failed went to panda3d. The model was just not sure what to do.

Reply

[-]

Virtamancer@reddit

ELI5 why doesn’t MLX need similar work to accommodate qwen3-next? Or does it? Do other/all formats require an update?

Reply

[-]

k_means_clusterfuck@reddit

day one llama.cpp support is not an absolute must. I'm happy they didnt wait to release Qwen3-next

Reply

[-]

shockwaverc13@reddit

who is this guy in the screenshot? why is he backseat moderating and claiming it'll take 2-3 months when he doesn't contribute anything to llama.cpp?

Reply

[-]

dizzydizzy@reddit

I guess we will know we have agi when it takes 5 mins instead of 2-3 months of engineer work

Reply

[-]

WithoutReason1729@reddit

Your post is getting popular and we just featured it on our Discord! [Come check it out!](https://discord.gg/PgFhZ8cnWW) You've also been given a special flair for your contribution. We appreciate your post! *I am a bot and this action was performed automatically.*

Reply

[-]