I generally agree that testing for user experience seems to be lacking or non-existent. My guess is that the labs are probably testing using their own internal dev code for both training *and* inference, and so the code they actually use to run benchmarks looks very different than what we see (HF chat template + runtime-specific chat parser).

It's not just Google though. It seems to be pretty pervasive in my experience. This rather depressing (still open!) issue shows that Meta can't even seem to agree on what the right format is, and reproducing benchmark results was not possible.

And there's another wrinkle. The model developers generally have nothing to do with quants/GGUFs. So it's understandable they don't test those. It is kind of surprising when some of the models are for "edge devices" like Gemma... you would think they would want to quantize themselves.

I think the most frustrating thing about all of this is that it's so difficult to know whether a malfunctioning model is simply a poor/weak model, or something else: quantization problem, runtime problem, chat template problem, or some other subtle issue? I was thinking the other day that it's kind of crazy that LLMs don't ship with test suites. Something like pairs of OpenAI API request and response pairs. Obviously sampling and quantization change responses, but at least it would be a way to record the "expected behavior" for context. Instead the best option is to post on reddit and see if other people are having the same problem!