LLM inference is memory-bandwidth bound during token generation. The formula is simple:

tokens/sec ≈ memory bandwidth (GB/s) ÷ model size (GB)

So a 70B Q4 model (~40 GB) on M4 Max: ~546/40 ≈ 13-14 tok/s theoretical max (real-world: 11-12 tok/s)

Massive headroom = unified memory architecture lets you load models that won't fit on consumer GPUs (70B+ on 64-128GB Macs vs. 24GB VRAM limits on RTX 4090).

MoE speeds things up via sparse activation:

• Dense model: Every token passes through all parameters
• MoE model: Every token activates only ~2-8% of parameters

On Apple Silicon:

1. Fewer weights to fetch per token results in less memory bandwidth consumed
2. Same model quality, lower inference cost results in higher tokens/sec
3. Can run larger total parameter counts while staying within bandwidth budget

Apple Silicon's GPU has decent compute but can't match high-end NVIDIA GPUs (H100, A100)

MoE helps here since fewer FLOPs per token during generation

Basically Apple Silicon's unified memory architecture provides exceptional bandwidth which is the primary bottleneck for LLM inference. However, token generation remains bandwidth-limited, capping speeds at 15 tok/s for dense 70B models. Mixture of Experts architectures dramatically improve this by activating only 2-10% of parameters per token, effectively reducing the bandwidth requirement and enabling faster inference or allowing larger models to run at the same speed.