Warning, info dump essay incoming

Yeah, basically. The key requirement is that the draft model and target model share the same tokenizer, since the draft has to produce token IDs that the target understands. Same model family is the easiest way to guarantee that — Qwen draft + Qwen target works, Gemma draft + Gemma target works, but Qwen draft + Llama target won't because the vocabularies differ.

Quick clarification on terms because I went down this rabbit hole myself recently--

Speculative decoding is the general technique — small fast "draft" model proposes N tokens, big "target" model verifies them all in one parallel pass. If the draft was right, you accept those tokens at the speed of one forward pass instead of N. Already in llama.cpp via --draft-model. Basically, the small model is writing the essay FAST, and shows the large model it's homework so they can cheat on the exam. For each part, the large model either says "yep, that's what I would've written" and keeps it, or "nope" and writes the rest itself starting from the rejection point. The large model does this, then turns it in to the teacher (you). The end result the teacher sees is that the large model turned in a pretty good exam, and did it faster than he usually would have. It was slower than what the small model did, but largely more accurate and informative because it only kept the parts that made sense.The large model balanced speed, efficiency, and accuracy.

MTP (multi-token prediction) in the strict sense is an architectural feature where the model has multiple prediction heads built in (DeepSeek V3 popularized this). Google's recent Gemma 4 announcement uses "MTP" loosely — what they actually released is small drafter models for classical speculative decoding, not built-in heads.

On the Gemma E2B/E4B side: those are dense models, not MoE. The "E" stands for Edge (Google's edge model family). E2B is ~2B params, E4B is ~4B params, all parameters active per token. These should be straightforward speculative decoding targets. It is really important that they release these, because everyone has been waiting on it. They teased it a few weeks ago when they showed benchmarks using this "MTP" method, and a lot of people found themselves a bit disappointed at the speed.

One important thing I discovered:

On Qwen3.6-35B-A3B: it's MoE — 35B total params, ~3B active per token. The router selects 8 of 256 experts per token. Speculative decoding still works on MoE, but the gain is somewhat smaller than on dense models. When the target verifies N draft tokens in parallel, those tokens may route to different experts, so the weight-load amortization that makes spec decoding fast is partial rather than complete.

For the smaller-Qwen-as-drafter idea (Qwen3.5-0.8B drafting for 3.6-35B-A3B): tokenizer compatibility is the first thing to check. If those two share vocabulary, it should work. Acceptance rate will determine actual speedup — could be anywhere from 1.2x to 2x depending on how well the small model predicts the big one's distribution. Theory says spec decoding on MoE just trims the win because parallel verification doesn't amortize as cleanly. In practice on my hardware, it was a 3× regression (10.5 tok/s vs 30 tok/s baseline) even with 100% acceptance rate using same-family Q4 target + Q2 draft. Your mileage really does depend on whether your hardware is compute-bound or bandwidth-bound.

BUT!! Here's the big caveats: using .8B as a drafter for a much better model is certainly going to give you only a very small increase ~10-20%. For drafting to work, the small model needs to get a decent amount of the information correct. .8B isn't really gonna cut it. Also, spec decoding is just a lot less efficient on MoE models. Unless Qwen releases a model specifically for drafting for their MoE model, or their 27B dense model, you might not find a huge jump. Or, you could I guess. The more I mess with this stuff the less I think I understand lol. Everything depends upon whether your setup is compute bound or bandwidth bound. Once you know which you fall under, predicting gains becomes a lot easier.

If you want to test it: --model-draft is the flag. Watch acceptance rate in the server logs. If acceptance is high but your tok/s is lower than target-alone, you've hit the same wall I did.