MiniCPM4: 7x decoding speed than Qwen3-8B
Posted by Lynncc6@reddit | LocalLLaMA | View on Reddit | 34 comments
MiniCPM 4 is an extremely efficient edge-side large model that has undergone efficient optimization across four dimensions: model architecture, learning algorithms, training data, and inference systems, achieving ultimate efficiency improvements.
- 🏗️ Efficient Model Architecture:
- InfLLM v2 -- Trainable Sparse Attention Mechanism: Adopts a trainable sparse attention mechanism architecture where each token only needs to compute relevance with less than 5% of tokens in 128K long text processing, significantly reducing computational overhead for long texts
- 🧠 Efficient Learning Algorithms:
- Model Wind Tunnel 2.0 -- Efficient Predictable Scaling: Introduces scaling prediction methods for performance of downstream tasks, enabling more precise model training configuration search
- BitCPM -- Ultimate Ternary Quantization: Compresses model parameter bit-width to 3 values, achieving 90% extreme model bit-width reduction
- Efficient Training Engineering Optimization: Adopts FP8 low-precision computing technology combined with Multi-token Prediction training strategy
- 📚 High-Quality Training Data:
- UltraClean -- High-quality Pre-training Data Filtering and Generation: Builds iterative data cleaning strategies based on efficient data verification, open-sourcing high-quality Chinese and English pre-training dataset UltraFinweb
-
UltraChat v2 -- High-quality Supervised Fine-tuning Data Generation: Constructs large-scale high-quality supervised fine-tuning datasets covering multiple dimensions including knowledge-intensive data, reasoning-intensive data, instruction-following data, long text understanding data, and tool calling data
-
⚡ Efficient Inference and Deployment System:
- CPM.cu -- Lightweight and Efficient CUDA Inference Framework: Integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding.
- ArkInfer -- Cross-platform Deployment System: Supports efficient deployment across multiple backend environments, providing flexible cross-platform adaptation capabilities
LagOps91@reddit
I'm not too interested in small models as I am able to run larger models, but I am impressed with the results in terms of efficiency and architecture optimisation. Great work on this!
InsideYork@reddit
Why not? I think use case is the most important, if it have constraints on your usage then LLMs are not so spectacular, they’re a less efficient way to do programming tasks I’d have not done.
LagOps91@reddit
simply because i can run larger models at good speeds, so i default to using those.
InsideYork@reddit
Do you ever want faster speeds? How about use multiple at a time or use one for a specific reason such as types of queries?
LagOps91@reddit
sure, faster speeds are preferred. If i want something fast I use Qwen 3 30B 3A, that gets me 30-70 t/s depending on context. it's way faster than reading speed, even with reasoning and i'm not sure going any faster has use for me.
InsideYork@reddit
If you just need to ask a local AI 1-2 questions at a time you don’t need to use smaller models.
LagOps91@reddit
then i don't understand what you are trying to say.
InsideYork@reddit
Longer context windows matter if you aren’t only asking it 1-2 questions.
LagOps91@reddit
i still don't understand. isn't the point of this model to have good performance even with long context? And yeah, i'm having longer conversations. I run Q3 30b with the full 40k context.
InsideYork@reddit
I didn’t like it for what I tried it for, I found it very fast though.
Gemma is better with language than the Chinese models and even I used 4B I found it produced just as good outputs as 12B for the kind of questions I used it for but much faster. I use a speciality small LLM for medical questions as well for my 1-2 questions.
JustImmunity@reddit
When i want faster speeds, usually i can parallelize my questions in some capacity, so i just spin up VLLM and batch it all.
TopImaginary5996@reddit
Fruit for thought:
LagOps91@reddit
yeah of course, not saying anything against those points. I'm just saying that I am not trying out the huge mountain of small models, I have already quite a few large models to try out.
in the end, it's quite unlikely that a small model would outperform models 3-4x the size, so i'm just not running them. I am not interested in running multiple models at the same time - at least not text models. But a text model and an image model... that's something worth considering.
Of course, the research done on smaller models is valuable! I'm not saying it's not! I'm quite excited for any advances made and I'm waiting for larger models to adapt some of these ideas.
NeuralNakama@reddit
It's really much much faster, but I wonder if we could be even faster if we quantize with Unsloth.
power97992@reddit
Does it run on ollama or Lm studio? Quality is more important than speed.. How is the quality?
lly0571@reddit
Minicpm 4 is more like GLM4-0414-9B and both of the have a 32(2) GQA config. The model(
MiniCPM4-8B-marlin-vLLM+MiniCPM4-8B-marlin-Eagle-vLLM) is likely 30-40% faster than Qwen3-8B-AWQ + Qwen3-0.6B under low-load conditions.Minicpm:
Qwen
Every-Comment5473@reddit
Will MLX possible?
foldl-li@reddit
Day 1 support from chatllm.cpp.
foldl-li@reddit
sparse attention is not implemented yet, but let's wait.
popegonzalo@reddit
This model's answer quality is pretty weak compared even to qwen3-0.6b.
Healthy-Nebula-3603@reddit
0.5b and is better than any 1b model ? Impressive
Lynncc6@reddit (OP)
they even have an 8B MLLM on par with GPT-4o
ROOFisonFIRE_usa@reddit
doubt.
Chromix_@reddit
I'd really like to see how this performs in the fiction.liveBench test - how much connection between snippets of information are lost due to that.
Regarding the benchmark speed: Qwen3-8B-UD-Q8_K_XL runs at about 120 t/s on a 4090. So that benchmark was likely run with FP16 (or BF16?) models. Thus, the to be expected speed is even higher with a Q4 quant.
ElectronSpiderwort@reddit
Say friend, what has been your experience with Q8_K_XL vs. Q8.0? Do you get any significant performance difference for that extra 24% space it is using?
Chromix_@reddit
Oh, there is a difference, my room gets slightly warmer when using it 😉.
In extended practical tests I find it difficult to see a clear difference between a Q6_K and the Q8_L_XL. It probably exists in some way, but probably isn't worth the extra compute.
tralalala2137@reddit
That decoding speed is crazy fast on RTX 4090. Wonder if this will eventually come to llama.cpp.
no-adz@reddit
Last one did: [2024.09.16] llama.cpp now officially supports MiniCPM3-4B! [GGUF Model | Usage]
tralalala2137@reddit
We can not stop winning.
smahs9@reddit
They have also released an inference runtime with sparse attention, custom speculative sampling and marlin matmul kernels (they even released a marlin optimized quant). It will be unlikely for llama.cpp to replicate those results. I would be curios to see how far vllm goes.
Calcidiol@reddit
Thanks OpenBMB-MiniCPM, good work!
I am curious to experiment and see how the efficient attention and fast decoding implementations / speculative decoding etc. can combine to enable a fast agentic model that can quickly process input / output data in some pipeline.
AaronFeng47@reddit
When gguf?
no-adz@reddit
I think it will come, like the last one: [2024.09.16] llama.cpp now officially supports MiniCPM3-4B! [GGUF Model | Usage]
DeltaSqueezer@reddit
This is a very interesting release! There's so much here. Thanks for sharing.