I genuinely can't tell if this is propaganda. The article talks about this like it's like some universal phenomenon, as if polish is just a universally "better" language. And it's written by a polish journalist. I also find it funny how he included DeepSeek in the list of models they evaluated, even though the researchers just use it as an example in the addendum, not doing the full benchmark for cost cutting purposes. (lmao)

The benchmark that's being evaluated in the paper is RULER, which is composed mainly of NIAH and retrieval tasks. If we have a language like polish in the data that tokenizes poorly, a single key will likely be spread out over multiple tokens, meaning it's easier for the model to see that all of the tokens match or don't match, compared to the significantly more clumped distributions of English or Chinese. Now yes, they do try to control for it, but I'd call their approach bullshit. Two tokens being attended to in a single word for a task like NIAH makes a massive difference than in general having 100,001 tokens in context instead of 100,000.

It's also important to mention that all the multilingual versions of RULER they made here are evaluated on completely different texts. Some of them with different formatting (see German vs English, for example). And they only seem to use one real text per language for NIAH. So it'll never be an apples-to-apples comparison unless we somehow same a million diverse texts that never ever appear in the training data for every variant (which, for smaller language groups, is hard!)

The model choice here is also questionable, they sample all of... six. Qwen 2.5 7B, Qwen 2.5 72B, Llama 3.1 8B, Llama 3.3 70B, o3-mini-high, and gemini-1.5-flash. Funnily enough, you can see that models with better tokenizers (specifically, Gemini) don't really have this bias. They have their own, different biases: o3 mini, besides polish, also really likes Norwegian, and Gemini REALLY likes Sesotho, despite it being an insanely undersampled language. But all the characters there are latin (meaning the model can steal understanding from bigger datasets, like english), and the model's likely been trained on all available public data, so it's insanely dubious to claim that this somehow makes it a "better" language to prompt with.

This doesn't even pass as good ragebait. Come on poland, do better.