Likely 90% of startup hype.
- There were sparse attention systems before, such as Google BigBird (not generative LLM, but more like sparse attention BERT) - somewhat better, but not enough to become industry standard. Also current LLM have positional embeddings that prioritize close tokens strongly.

- The most expensive calculation in attention is vector projection which is O(N). Calculating many cross product before softmax is O(N\^2) but ultimately it is not expensive as matrices are not large. The problem, of course, happens with decoding and KV caches as you need to store these projections, but for input context it matters not.

- Therefore, sparse attention seems to be decent idea, but not genius solution and with tradeoffs.

- Real problem is not making 12M context, but make abstractive reasoning work reliably at like 50k context https://arxiv.org/abs/2502.05167 and also make LLM not break randomly if you feed it with lots of irrelevant details https://machinelearning.apple.com/research/illusion-of-thinking

- Do not believe startups in general until they show reproducible result. For my space of interest (GUI Agents) there are many startups which show solutions that obviously don't work well and will not work well (run Claude or GPT with few agentic prompts) and yet show off benchmark scores like 90% accuracy on very complex tasks.