Qwen 3.5 MXFP4 quants are coming - confirmed by Junyang Lin

[-]

coder543@reddit

That tweet doesn't say anything about doing QAT to get the MXFP4 quants, just releasing some MXFP4 quants.

Reply

[-]

HareMayor@reddit

Isn't the nvfp4 supposed to be a way better thing? Why aren't companies putting resources towards that instead?

Reply

[-]

NVFP4 is based on MXFP4, but NVFP4 is proprietary and poorly supported (in terms of hardware platforms *and* Nvidia's own software/drivers/documentation). Why would companies put resources towards NVFP4? When Nvidia is ready, Nvidia will make it easier for companies, and then companies *might* consider adopting it.

Reply

[-]

Porespellar@reddit

Unless you consider DGX Spark, 5090, 6000 Pro as “consumer” platforms. NVFP4 should smoke pretty fast on those Blackwell chips if they ever get it working with vLLM…..in theory.

Reply

[-]

coder543@reddit

DGX Spark does not support NVFP4 period right now. There have been extensive discussions on the topic on the Nvidia forums. Nvidia's own support for Spark has been a joke.

Reply

[-]

halcyonhal@reddit

I’m using the latest nemotron nvfp4 on a 5090 and vLLM. It’s the 0.16.0 rc version (prerelease). Works great with cuda 12.9.

Reply

[-]

coder543@reddit

And how does the performance compare to MXFP4?

Reply

[-]

halcyonhal@reddit

I’ve not run mxfp4.

Reply

[-]

inevitabledeath3@reddit

I literally saw a video of someone running NVFP4 on spark not long ago. It was on Level 1 Techs.

Reply

[-]

reto-wyss@reddit

As far as I know vllm uses cutlass since the latest version for NVFP4, at least some of the "warnings" went away on Pro 6000 - haven't tested with a 5090 yet.

Reply

[-]

halcyonhal@reddit

Latest released build, 15.1 works for pro 6000 but not 5090. Prerelease 16.0 works for both.

Reply

[-]

dampflokfreund@reddit (OP)

Hmm true, but what would be the point of making regular quants when there's already bartowski and unsloth?

Reply

[-]

Klutzy-Snow8016@reddit

They usually release their own GGUF quants. Some people / companies feel more comfortable using an official 1st party repo.

Reply

[-]

LA_rent_Aficionado@reddit

No clue but it would make a little more sense than re-training the same model in 4 bit when they likely have a queue of 8B,14B,32,etc. models to train

Reply

[-]

JockY@reddit

From the blog they didn’t QAT but instead used an FP8 pipeline. Very cool to hear an official MXFP4 is coming because Qwen will obviously have the best possible calibration dataset.

Reply

[-]

fallingdowndizzyvr@reddit

> GPT-Oss release - they trained their model in 4 bit No they didn't. I was *post-trained* at 4 bits, training was not at 4 bits Straight from the horses mouth. "The models were **post-trained** with MXFP4 quantization of the MoE weights" https://huggingface.co/openai/gpt-oss-120b Unsloth has already made MXFP4 quants. https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF/tree/main/MXFP4_MOE So they aren't coming. They have already been here.

Reply

[-]

dampflokfreund@reddit (OP)

The MXFP4 quants by Unsloth are not QAT though. Completely different thing from what I am talking about.

Reply

[-]

fallingdowndizzyvr@reddit

And just because they make MXPF4 quants doesn't mean it'll be QAT. You are assuming they will be. Which is doubtful since QAT happens during training. So are you suggesting they are going to retrain their models?

Reply

[-]

chisleu@reddit

That's GGUF. Some of us (vllm/sglang/linux users) need AWQ

Reply

[-]

dinerburgeryum@reddit

The Gated Attention mechanism has a similar side-effect to that of Attention Sinks: it smooths out the wild activation for low-attention tokens and keeps the tensor values more consistent, making quantization less damaging. I don't think they'll bother doing any QAT; presumably they won't have to.

Reply

[-]

dampflokfreund@reddit (OP)

Interesting. But I assume post trained 4 bit quants would still fare a lot better than the usual Bartowski and Unsloth quants. But we simply have no data right now.

Reply

[-]

tarruda@reddit

I'm pretty amazed at how good these IQ2_XS quants were: https://huggingface.co/ubergarm/Qwen3.5-397B-A17B-GGUF/tree/main/smol-IQ2_XS This weekend I'm going to try running mmmlu and gpqa diamond to see how they compare with the official results.

Reply

[-]

dinerburgeryum@reddit

ubergarm at it again! 400B is a little too rich for my blood, but I'm excited to run some experiments with Coder-Next, which is built on the same architecture.

Reply

[-]

junyanglin610@reddit

no i mean fp8...

Reply

[-]

dampflokfreund@reddit (OP)

Oh, sorry about that. Would you like me to delete this post or edit it with the correct information?

Reply

[-]

PerPartes@reddit

To be clear, GPT OSS was just post-trained (aka fine-tuned) in MXFP4, not fully trained. But the FP4 marketing was huge and who cares about details…

Reply

[-]

asklee-klawde@reddit

been running qwen2.5-coder:7b for all my dev work and it's wild how good it is at that size. native 4-bit on the bigger models is gonna be huge

Reply

[-]

Septerium@reddit

Nice try, outdated LLM

Reply

[-]

pulse77@reddit

I hope we will soon get NVFP4 quants in llama.cpp and in models. These are even better than MXFP4...

Reply

[-]

jinnyjuice@reddit

On Hugging Face, there are few people there that are really dedicated to NVFP4. If there is no model in NVFP4, they gladly create them through requests.

Reply

[-]

pulse77@reddit

But llama.cpp has no support for NVFP4 yet...

Reply

[-]

AciD1BuRN@reddit

Does llama support nvfp4 yet ? Thought it was still pending

Reply

[-]

dampflokfreund@reddit (OP)

But they are only supported by Blackwell GPUs.

Reply

[-]

pulse77@reddit

Even on pure CPU they would yield better quality than MXFP4 for the same size...

Reply

[-]

dampflokfreund@reddit (OP)

But also much slower for most people. It is just not worth it at the moment.

Reply

[-]

TheThoccnessMonster@reddit

Try to think forward a bit. Both are fine. One is more fine for less people, yes.

Reply

[-]

JockY@reddit

*fewer

Reply

[-]

MerePotato@reddit

Yeah, when we get a decent software kernel they'll be worth it but the tech just aint mature enough yet

Reply

[-]

Significant_Fig_7581@reddit

That!

Reply

[-]

skrshawk@reddit

I'm on MLX - how is OpenAI/Qwen doing a MXFP4 quant different than me doing it on my Mac Studio from the unquantized model?

Reply

[-]

JaatGuru@reddit

Which model are you using on your Mac Studio. I’ve similar structure and happy with gpt-oss-120b q4.

Reply

[-]

skrshawk@reddit

GLM-Air 4.5 and Largestral on MXFP4 quants primarily. Q4 of some of the ones I really want to use in that quant are a very tight fit.

Reply

[-]

NeuroVexon@reddit

This is huge. Native quantization vs post-training quants is a massive quality difference. Really hope more model providers go this route. The GPT-O approach showed it works, glad to see Qwen following.

Reply

[-]

Significant_Fig_7581@reddit

Nothing is going to make my day like a Qwen 35B in MXFP4 that could crush GLM 4.7 Flash, And after that GLM 5 20B OSS or something that could crush this Qwen model, I'm Daydreaming....

Reply

[-]

tiffanytrashcan@reddit

https://preview.redd.it/j29b2kq639kg1.jpeg?width=1080&format=pjpg&auto=webp&s=199b2c1b5e923ba04e8f25f69cb9b0435ee4e455

Reply

[-]

Significant_Fig_7581@reddit

Lol, I forgot to say it'd be even more interesting if the imaginary GLM 5 20B OSS MOE is natively trained using NVFP4 not MXFP4 and support to NVFP4 is also added to llama.cpp....

Reply

[-]

Consistent-Height-75@reddit

And then a new Bert-Base-2026-100M crushing Opus 4.6 and we can run it on refrigerator.

Reply

[-]

chensium@reddit

Oh man fridge ai rigs are the next frontier! Already on appliance-rated outlets, with built-in cooling. Philharmonic 4.9 be like "If you walk to the carwash make sure you pick up some milk on the way back because we're down to the last 4 gallons"

Reply

[-]

pkmxtw@reddit

Nice! That is just about the right size for the Q0.1 quant to fit this opus 4.6 killer on my floppy disk!

Reply

[-]

Significant_Fig_7581@reddit

You are delusional.

Reply

[-]

jonydevidson@reddit

In 1 year we will have 30B models performing like the current frontier models. No need to dream, just wait. Chill out, go play some games, run a few hikes, do some trips, come back in a few months, itll be a different world already.

Reply

[-]

TheTerrasque@reddit

Nah. llama3 70b is still better than any 30b model for my use case *(Story writing / storytelling / rp)*. These days big MoE's provide better overall results though.

Reply

[-]

DeepOrangeSky@reddit

Yea, it seems like there is some sort of tipping point somewhere between 30b and 70b where it starts having enough parameters where the amount of world knowledge it can hold becomes enough that it starts feeling a lot smarter for prose-writing types of tasks. Also, the Llama 70b is a dense model, which seem like they tend to be better for writing than most of the similarly sized MoE models. Or at least traditionally that was the case, maybe finally starting to change as the MoEs continue improving (albeit mostly for coding and not as much for writing, but still some improvement spills over sometimes I guess). Then again Gemma3 27b feels bigger in terms of world knowledge/general smarts than it should, sometimes (seems like Google, being Google and all, knows how to really max that out), and be pretty good for writing for a model of that size, or at least some of its fine tunes anyway. So, who knows. I don't think Google will release a Gemma4 70b dense model, but, if they did, I bet it would be a friggin monster at writing. Probably would be about on par with models 10x its size at writing, or better than even those, maybe.

Reply

[-]

TheTerrasque@reddit

> Also, the Llama 70b is a dense model, which seem like they tend to be better for writing than most of the similarly sized MoE models. GLM-4.6 and STEP-3.5-Flash have been the notable ones for me lately, 357B and 196B params respectively. Notably larger overall, but the MoE architecture still lets it run noticeably faster on my system than the dense 70b+ models.

Reply

[-]

DeepOrangeSky@reddit

GLM is out of reach for me. Step-3.5 is frustratingly close for me though (I have a mac with 128 gigs), so I might try it on some low quant, just to see, but, so far the Mistral 123b and fine tunes are the strongest I've used for writing tests locally, by quite a bit. It would be fun to have enough ram for some of the really big models though. I decided that since 128gb could run basically all the fine-tune sized models (123b and smaller) at q4 or higher, that I would start with that size of mac, and then if I decide a few months from now, once I am more experienced and know what I'm doing more, and maybe also get more into coding and so on, that I still really want to be able to run the really huge models, then maybe I'll get whatever monstrosity the m5 ultra/maxed out big ram mac model is going to be this summer. But, I think I'm probably going to avoid spending more on gear between now and then, and just enjoy trying out and reviewing a bunch of fine tunes between now and then, and maybe trying to learn enough about coding to be able to actually have some slight idea of what I'm doing with the coding models by the summer time hopefully. Or Grok 5 brings AGI by then, and I upload my consciousness to mindtravel around VirtualAndromeda v2.1. Or everyone on earth gets fired, and we all go broke, and society collapses, and I eat some cans of Spaghettios on a rooftop while watching the sun set over the smoking ruins of some city like I'm in The Walking Dead. Not sure which of those main flow-chart branches it'll end up being, but those seem like the main ones for now. Probably geeking out over some expensive m5 ultra being the main one. But I bought a lot of Spaghettios, too, just in case.

Reply

[-]

Significant_Fig_7581@reddit

I agree really, But i want something to work with now, GLM is great! But I generally prefer talking to Qwen models, and I'm positive Qwen is gonna release the 35B MOE soon... So that's what I'm waiting for in the mean time...

Reply

[-]

jinnyjuice@reddit

GLM 5 Flash!

Reply

[-]

Significant_Fig_7581@reddit

I hope it'd come at a small size but I really can't see it being less than 80B...

Reply

[-]

Sabin_Stargem@reddit

Personally, I am hoping for a distillation at 80b-120b parameters. My meager gaming machine can't handle even a quarter of the biggest Qwen.

Reply

[-]

jacek2023@reddit

but first let's see smaller models (35B etc)

Reply

[-]

dampflokfreund@reddit (OP)

Yes these will come too. Maybe even in MXFP4 already. These smaller MoEs are very sensible to quants, so having native 4 bit quants is really awesome.

Reply

[-]

Overall-Somewhere760@reddit

Do you know when they release the 35B?

Reply

[-]

Dramatic_Spirit_8436@reddit

after that GLM 5 20B OSS

Reply

[-]

-dysangel-@reddit

Ah ok, I didn't realise that the unsloth ones wouldn't be as high quality - I went straight for mxfp4

Reply

[-]

audioen@reddit

MXFP4 is a decent quantization, but not the best. For example, with MiniMax-2.5, using it resulted in this perplexity: [https://huggingface.co/ubergarm/MiniMax-M2.5-GGUF/blob/main/images/perplexity.png](https://huggingface.co/ubergarm/MiniMax-M2.5-GGUF/blob/main/images/perplexity.png) showing that it is worse than IQ4\_XS which is also slightly smaller, so loss in both fronts at least in case of that model. It may be faster to infer, depending on hardware. We don't always have the MXFP4 results because people who quantize and systematically evaluate models are rare and even this came due to request, someone asked for it and ubergarm provided it. Still, anyone can run the wikitest and generate comparable results, it just requires download and the lengthy process of measuring the perplexity, perhaps 30 minutes per these larger models on a Strix Halo.

Reply

[-]

Professional-Bear857@reddit

I think a lot of the reason for the good performance of mxfp4 Vs others is that it doesn't quantize down the other model layers and only quantises the experts into mxfp4, other quants should do the same, but they often reduce important layers down to q5 or q6 which I think harms the performance.

Reply

[-]

dampflokfreund@reddit (OP)

No no, the quality benefit only occurs when the model is trained in 4 bit. So far that only applies to Gemma 3 QAT and GPT-Oss in mxfp4. For non 4 bit models I think the Unsloth UD\_Q4\_K\_XL are higher quality than mxfp4 but not sure.

Reply

[-]

Septerium@reddit

I think Kimi 2.5 is trained in 4-bit too. This is why I think it is a much more interesting model compared to freaking behemoth GLM-5, which is bigger than 1.5TB in its true form

Reply

[-]

sleepingsysadmin@reddit

Ive watched a few videos on quants. Unsloth's Q4\_K\_XL has been a go-to for me. When it comes to a 35B model, i do think unsloth's will still be superior.

Reply

[-]

dampflokfreund@reddit (OP)

No, if a model is trained in 4 bit and released as a mxfp4 or QAT quant, they are far superior to Unsloth's quants.

Reply

Reply to Post

70 Comments