Yup, I've been liking both Olmo-3.1-32B and K2-V2 for future community-driven projects.

Both are somewhat under-trained (189 tokens/parameter for Olmo-3.1-32B, and 112 tokens/parameter for K2-V2) but also hit above their weight in terms of overall inference quality. This means they should be strong base models for further training, and they should be able to absorb a lot more training data without "overcooking".

They are not without drawbacks, though. Neither are great for creative writing; Olmo-3.1-32B-Instruct really shines at STEM tasks, while K2-V2-Instruct is great for analysis, RAG, and logical problem-solving.

However, when using Olmo-3.1-32B-Instruct for real-world physics problems I found it had some odd gaps in its knowledge (most glaringly, it insisted that Lithium-6 fission was not physically possible).

K2-V2-Instruct's achilles heel is that it gets very, very slow as context grows large (I think due to its simple architecture; it is using ye olde "llama" design). Its 226K token context inference is only 6% as fast as it is at 4K token context.

It's very likely that Olmo's shortcomings might be covercome with more training, while K2-V2 should be fine at lower-context-length inference, so I wouldn't rule them out.

Also, the main reason both of these models exhibit such high quality inference is because of the very high quality data used for training them, all of which is available for download on Huggingface. Even if we do not use the models themselves, using their data for community training projects seems like a gimme.

Another approach I've been doodling with recently is using Gemma-4-31B-it as a base model for an MoE. Since Google switched from their old, bad Gemma license to Apache-2.0, Gemma 4 is viable in ways the older Gemma models were not.

I had thought AllenAI's FlexOlmo technology was the answer to our prayers, in that it would allow us to federate community training without getting hung up on intercommunication during training, but u/mz_gt pointed out shortcomings of FlexOlmo which makes it a lot less viable for that than I'd hoped.

Despite that, I think the same general approach to federated MoE training might work, if we can figure out how to keep the experts intercompatible without synchronizing during training.

One way we might be able to do that is to limit the number of sparse layers, keep the rest of the model dense, and bucketize the experts post-training so that mutually compatible experts are grouped. The routing logic (which unfortunately would need to happen post-merge; FlexOlmo really fell down there) would then choose only experts within those groups.

I need to do some groundwork to ascertain if this approach is viable at all, though. It might prove to be a non-starter.

If something like that can work, though, it would mean we could federate the training of experts, using Gemma-4-31B-it as the "anchor" model. Each participant would add training only to the middle six-layer "block" (five sliding window attention layers + one full attention layer) which RYS theory predicts is structure-agnostic, collectively making a 2.9B parameter expert.

The end result would be a model with 28B of shared/dense parameters and some number of 2.9B parameter expert blocks. With 32 experts, four chosen, that would give us a MoE with 121B total parameters and 40B active parameters.

It looks good "on paper" but like I said there's a lot of groundwork to do first to see if the approach is even viable. Nonetheless, I'm confident that the open LLM community will have options, moving forward, if we need to pick up the mantle of open LLM R&D ourselves.