Last year researchers affiliated with NVIDIA, University of Warsaw, and University of Edinburgh published Dynamic Memory Sparsification (DMS), a KV-cache sparsification technique using learned per-head token eviction, reporting up to 8x KV-cache compression.

I found the results intriguing to build a small reference implementation and trainer to sanity-check the idea. On WikiText-2 with Llama 3.2 1B, I was able to get a rough replication:

Training the DMS predictors took about 20 minutes on the PRO 6000 and the compression looked basically lossless. One small problem though, my HF reference implementation ran at about... 18 tok/s.

So, after a few weeks of kernel grinding, I'm pleased to announce FastDMS, an MIT-licensed implementation of DMS with compact KV storage that physically reclaims evicted slots. It is tested on NVIDIA's original Qwen 3 8B DMS checkpoint as well as my own Llama 3.2 1B DMS checkpoint. (the original HF reference version and my trainer are in the repo as well): https://github.com/shisa-ai/FastDMS

On my benchmark setup, FastDMS uses 5-8x less KV memory than vLLM BF16 KV at 8K context while also decoding 1.5-2X faster than vLLM.

Compact DMS saves real allocator/device memory, not just theoretical KV bytes. The table below uses ctx_len=8192, gen_len=128. All vLLM baselines use exact-sized token pools matching the workload. KV/stage memory is the cache or cache-plus-staging footprint. vLLM BF16 means dtype=bfloat16 with kv_cache_dtype=auto; vLLM FP8 means kv_cache_dtype=fp8.

For those that are curious, yes, this beats out TurboQuant in both speed and memory usage:

Of course, none of this matters if the compression tanks output quality. In theory, DMS eviction is applied before FP8 quantization, deciding which tokens to keep or evict, so the quality comparison for FastDMS compact-DMS should be the same versus FP8 quantization alone, but it's still worth double-checking quality.

This is measured by generating tokens with a compressed KV cache and comparing against an uncompressed reference, token by token. Lower KLD (KL divergence) is better - it means the compressed model's next-token probabilities are closer to the reference. Higher token match is better - it means greedy decoding produces the same output.

How to read the columns:

• KLD vs ref - KL divergence in nats/token between the compressed and reference logits. Measures how much the probability distribution over next tokens shifts due to compression. Lower is better; 0.000 means identical.
• Token match - percentage of greedy-decoded tokens that are identical to the reference. 96.9% means ~2 out of 64 tokens differed.
• Tokens scored - how many decode steps could be compared. Once the candidate produces a different token than the reference, the sequences diverge and later steps aren't comparable. 33/60 means quality metrics only cover the first 33 tokens before divergence - the reported KLD and PPL are over that prefix, not the full generation. A higher ratio means the comparison is more complete.

Test setup: ctx_len=1024, decode_len=16, four prompts (60-64 total decode steps). vLLM rows compare against vLLM BF16 full-KV logits. FastDMS rows compare against FastDMS with eviction disabled (reference window of 1M tokens, effectively keeping the full KV cache).

shisa-ai/Llama-3.2-1B-DMS-8x

nvidia/Qwen3-8B-DMS-8x

FastDMS compact-DMS scores 64/64 tokens on both models - every decode step was comparable to the reference, and the KLD is lower than or comparable to vLLM's own FP8 and TurboQuant compression. Note that PPL values across rows are not directly comparable when Tokens scored differs, because each row's PPL is computed over a different-length prefix.

What's the catch?

So, if this is so darn great, why wasn't everyone using it already? Well, it turns out if you want to implement this in a production engine like vLLM, you have to do major surgery to it. DMS compact KV touches nearly every serving-engine subsystem:

God bless anyone that wants to give this a swing. The kvcache compression seems real, and with a correct implementation there's no quality hit, and as shown by the FastDMS implementation, it looks like can run faster than non-DMS inferencing.

(lots more perf benchmarks, comparisons, and raw logs in the repo for those interested)