GLM-4.7-Flash soon?

Posted by rerri@reddit | LocalLLaMA | View on Reddit | 23 comments

I noticed a GLM-4.7 collection update which includes a hidden item and started digging a little. Looks like Zai is preparing for a GLM-4.7-Flash release: [https://github.com/zRzRzRzRzRzRzR/vllm/commit/872df7369f8d966f2b73596ea06787d893431a23](https://github.com/zRzRzRzRzRzRzR/vllm/commit/872df7369f8d966f2b73596ea06787d893431a23) https://preview.redd.it/pu0rf6jyuaeg1.png?width=450&format=png&auto=webp&s=95f85a9c48dfde58a0232b3a40991b3e899e4687

Reply to Post

23 Comments

[-]

Few_Painter_5588@reddit

They have a Flash model already, [https://huggingface.co/zai-org/GLM-4.6V-Flash](https://huggingface.co/zai-org/GLM-4.6V-Flash) So probably around 10B I'd assume. Their 4.6V Flash Model was decent.

[-]

botonakis@reddit

[https://huggingface.co/zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)

[-]

rerri@reddit (OP)

That's a dense model though. GLM-4.7-Flash is a MoE model.

[-]

Few_Painter_5588@reddit

Oh, good catch. There's not much context to go off. If I had to hazard a few theories, it's either the AIR rebranded or it's a 20-30B moe.

[-]

Resident_Suit_9916@reddit

https://preview.redd.it/ddpeu3s3ebeg1.png?width=441&format=png&auto=webp&s=2195b01b3ff72a003e4640ea875fc5a9b6202997

[-]

rerri@reddit (OP)

Yea, they updated their docs with a benchmark comparison to Qwen 30B and GPT-OSS 20B so probably close to them in size.

[-]

FullOf_Bad_Ideas@reddit

[-]

guiopen@reddit

Strange, they said in an AMA that they do not see value in MoE below 30b parameters, I think these might be dummy values

[-]

FullOf_Bad_Ideas@reddit

It's 29.27B A3B model config. I plugged a wrong number in my calculator for "Num MoE Layers" - I plugged in 1 instead of 46 and this twisted the results. Sorry.

[-]

rerri@reddit (OP)

Their docs include a benchmark graph with comparisons to GPT-OSS 20B and Qwen 30B + this snippet of text: >In mainstream benchmarks like SWE-bench Verified and τ²-Bench, GLM-4.7-Flash achieves open-source SOTA scores **among models of comparable size**. Additionally, **compared to similarly sized models**, GLM-4.7-Flash demonstrates superior frontend and backend development capabilities. 5B doesn't seem plausible.

[-]

FullOf_Bad_Ideas@reddit

It's 29.27B A3B model config. I plugged a wrong number in my calculator for "Num MoE Layers" earlier - I plugged in 1 instead of 46 and this twisted the results. Sorry.

[-]

lly0571@reddit

The model seems to be Qwen3-30B sized, with MTP and deepseek-V3 styled MLA, intresting.

[-]

FullOf_Bad_Ideas@reddit

thanks for correcting me. It's 29.27B A3B model config. I plugged a wrong number in my calculator for "Num MoE Layers" - I plugged in 1 instead of 46 and this twisted the results. Sorry.

[-]

Maximum@reddit

GLM 4.7 makes an excellent agent. If the flash stays excellent at the tool calling is will replace some of my sub agents with it and save a ton of tokens.

[-]

SlowFail2433@reddit

Tau bench and browsecomp scores promising

[-]

Odd-Ordinary-5922@reddit

moe model using a new architecture maybe

[-]

Wooden-Deer-1276@reddit

**Total Parameter Count (Storage Size)** Sum of all weights stored in memory. $$\\text{Total} = \\text{Embeds} + L \\times (\\text{Attn} + \\text{FFN}\_{\\text{total}})$$ $$\\text{Total} = 634.4\\text{M} + 47 \\times (24.25\\text{M} + 666.9\\text{M})$$ $$\\text{Total} = 634.4\\text{M} + 32,484\\text{M} \\approx \\mathbf{33.12 \\text{ Billion}}$$ **Activated Parameter Count (Inference Cost)** Sum of weights used to process a single token (forward pass). * **Active FFN:** Uses the **1 Shared Expert** \+ **4 Routed Experts** (`num_experts_per_tok`). * Active FFN Size = Shared (62.9M) + $4 \\times$ Routed (9.44M) = **100.66 Million**. * **Active Layer:** Attn (24.25M) + Active FFN (100.66M) + Norms/Router $\\approx$ **125 Million**. $$\\text{Activated} = \\text{Embeds} + L \\times \\text{Active Layer}$$ $$\\text{Activated} = 634.4\\text{M} + 47 \\times 125\\text{M}$$ $$\\text{Activated} = 634.4\\text{M} + 5,875\\text{M} \\approx \\mathbf{6.51 \\text{ Billion}}$$ # Summary Table |**Metric**|**Count**| |:-|:-| |**Total Parameters**|**\~33.1 Billion**| |**Activated Parameters**|**\~6.5 Billion**| *Note: This assumes all 47 layers follow the MoE structure described. DeepSeek models sometimes replace the first few layers with dense layers, but as that configuration is not visible in the snippet, the calculation strictly follows the visible MoE definitions.*

[-]

Toooooool@reddit

With how 4.6 was 106B and 4.6-Flash was 10B, think they'll make it 10% the size this time too? A \~35B LLM would fit nicely in 2x24GB setups

[-]

Environmental-Metal9@reddit

Wasn’t 4.6 357b params? 106b sounds like a REAPed version of it. 357B without quantization would require a full 8x H200/b200 node, like over 700Gb of vram required. Quantized you could get it way down, and reaped and quantized even more, probably down to a single card if you didn’t want to offload any layers

[-]

jacek2023@reddit

quarterly reminder https://preview.redd.it/jnhyinehzaeg1.png?width=948&format=png&auto=webp&s=0a1e1f7178feb8682da9528c4089b5d485947802 [https://huggingface.co/collections/zai-org/glm-46](https://huggingface.co/collections/zai-org/glm-46)

[-]

jacek2023@reddit

I am a simple man, I just want some Air

[-]

Familiar_Passage_861@reddit

Nice detective work, that commit definitely looks like they're gearing up for something big

[-]

rerri@reddit (OP)

Not sure if sarcastic since that's just a name change, but there's more to be seen in these branches to be seen for the code literate (which I'm not): [https://github.com/zRzRzRzRzRzRzR/transformers/tree/glm](https://github.com/zRzRzRzRzRzRzR/transformers/tree/glm) [https://github.com/zRzRzRzRzRzRzR/vllm/tree/glm](https://github.com/zRzRzRzRzRzRzR/vllm/tree/glm)