Qwen 3.5 27B - quantize KV cache or not?

Please note that my comment was written months ago, so things changed drastically since then. If you are using recent llama.cpp, I am assuming you are rotating your activations to reduce quality loss from quantizing the cache, so you will be getting better results than at the time when I wrote my comment. Important to understand that quantization of the model itself and of its cache are completely separate from each other. Also, not all models have quants higher than Q4, for example for Kimi K2.6 Q4\_X is the highest possible precision since it is the one that most accurately preserves the original INT4 quality in the GGUF format. The best possible quality for cache is F32, followed by BF16, and then F16 and lower. F16 works very well in most cases, even though there were rare exceptions where BF16 or F32 work better, but generally, there are no further improvement beyond F16 with most models, which also well supported on most GPUs (unlike BF16 for example), I guess this is why F16 is the default. I am still using Kimi K2.6 with F16 cache because not only it reduces possibility of quantization errors but also gives me a bit more performance than if I put more layers on GPUs. As of measurements with different cache quantization levels, the latest measurements that are up to date can be found in the patch that introduced the rotation: [https://github.com/ggml-org/llama.cpp/pull/21038](https://github.com/ggml-org/llama.cpp/pull/21038) (which means that any tests for different quantization levels you may find that predate merge of this patch are out of date).