TurboQuant itself is a quantization method like so many others before it, and if you're willing to sacrifice speed and simplicity for memory savings it lets you do that in a slightly new way. But we've had "lossless 2-bit KV cache" in various forms for years, and it never gains traction because the tradeoffs just aren't worth it. Still, it's an interesting bit of research with a few novel ideas worth integrating.

The real issue is with the blog post making claims like "lossless", "zero overhead" and "8x faster." There's no source for any of those claims. The paper doesn't mention anything about TQ being faster (except compared to CPU-based RaBitQ in a semantic-search context), and the "zero overhead" seems to refer to distortion rates, not computational overhead.

There are also no real implementation details in the paper, just a snippet of pseudocode and some synthetic results. But the proposed method inherently adds a lot of computational overhead. It may still give you a net speedup in memory-bound situations, but that speedup isn't implied by the algorithm, isn't universal even if it can be achieved situationally, and is always going to be less than a simpler quantization scheme under the same circumstances.

So then it would come down to accuracy, right? But then why not compare it to other methods that make similar claims:

• GEAR: Combines quantization with low-rank and sparse matrices, "near-lossless" at 2 bits
• QAQ: Adjusts bitrate per token according to estimated importance
• MIKV: Aggressive quantization for most tokens, preserves "pivotal" tokens
• RotateKV: 2-bit method using rotation, "near-lossless"
• PM-KVQ: Specifically addresses long CoT contexts where many "near-lossless" methods turned out not to be so lossless in practice
• etc.

FP8 is commonly used in production, is trivial to implement and comes with immediate performance benefits. NVFP4 is the really interesting one because of its extremely high throughput on Blackwell GPUs, yet it still has a reported <1% accuracy loss on real benchmarks.

So even if TQ did outperform everything else, you should still curb your expectations somewhat: maybe you might reduce the effective size of your cache from 4 bits to 3.5 bits. For modern models that already employ a lot of memory-saving techniques at the architectural level (linear attention, MLA, SWA) it's simply not that big a deal.

So no, it's not revolutionary, and yes, Twitter is out of control. In Google's own (mind you, very limited) testing it doesn't even unambiguously outperform KIVI from 2024.