Mistral Medium 3.5 128B

I have not yet a chance to try this model, but generally MoE with a reasonably-sized shared weights/activated parameters has significant advantage over large dense as LLM activation is mostly noise, which is empirically just bad rather than something useful (naively upping # of active experts for existing MoE model simply make the model worse, low-pass filter type gating techniques work well with LLM, etc). The "partitioning" done by MoE architecture works to filter out those noises.

A remarkable advantage of large dense usually comes with large hidden dim (GPT-3 DaVinci was 175B with 12288 hidden dim IIRC? Llama-3 405B is 16384, Mistral Medium 3.5 is also 12288) which would be able to distinguish and partition extremely close features that would otherwise overlap in say a 5120 hidden dim model and (hopefully) worked through layers. That also means the (large hidden dim) model could place Paris and London (along with related things like Toulouse and Brighton) to the polar opposite end of latent space if there are enough evidence in the datasets to do so. I'm not sure if that is good or bad. Good old GPT-3 DaVinci had a feel that inference path diverges really hard (the model goes from one mode to another; in today's standard that would at least mean base/instruction/single-turn reasoning/terminal-agentic modes) by just one token.

For creativity and generalization of hard problems, one would generally want more layers rather than larger hidden dim within the same parameter count, unless hidden dim is too small to create meaningful basins anymore.