The architecture powering Granite 4.0-H-Micro, Granite 4.0-H-Tiny and Granite 4.0-H-Small combines Mamba-2 layers and conventional transformer blocks sequentially in a 9:1 ratio. Essentially, the Mamba-2 blocks efficiently process global context and periodically pass that contextual information through a transformer block that delivers a more nuanced parsing of local context through self-attention before passing it along to the next grouping of Mamba-2 layers.

Huh, this is kind of similar to Gemma and Gemini 1.5 in using a N:1 interleaving layers of dense attention along with something else, of course for Gemma, it was a local windowed attention transformer layer instead of an RNN layer, and at a more conservative 4-6:1 ratio. It's imo a great idea, the main performance bottleneck in Mamba is a breakdown of inductive reasoning without the dense attention, but it is only needed relatively sparsely to be able to develop the proper inductive biases to create those circuits. The quadratic bottleneck remains, so you'll still need a way to solve the quadratic communication overhead during training for long sequences, but it should be much cheaper to train now