KV-cache eviction controls decoding memory by removing stored key-value attention states while leaving the input prompt unchanged. It complements prompt compression: compression reduces prefill cost, whereas eviction reduces memory during autoregressive generation. section 13.8.2 section 13.0
Different eviction strategies make different compromises between simplicity, selectivity, and accuracy:
- H2O maintains a fixed-size cache and evicts tokens with low accumulated attention scores. section 13.8.2
- SnapKV performs one-shot pruning after prefill by retaining high-attention hit tokens before generation. section 13.8.2
- RazorAttention preserves full caches for retrieval heads, prunes non-retrieval heads, and uses a compensation token; it reduces cache size by roughly 70 percent without retraining. section 13.8.2
- ManifoldKV scores tokens by L2 distance from a local context centroid, preserving salient radial outliers such as names, dates, and numerical values. section 13.8.2
Needle retrieval under compression
On a three-key needle-in-a-haystack task at 50 percent compression, ManifoldKV outperformed cosine-based alternatives by 15.4 points. The reported geometric advantage remains promising rather than settled because the work is a February 2026 preprint and has not been peer-reviewed. section 13.8.2
Table 13.4 Method tradeoffs
|
Compression paradigm
|
Primary methodology
|
Optimal ratio
|
Compute overhead
|
Risk of semantic loss
|
|
Heuristic Pruning
|
TF-IDF Filtering
|
Low (~2x)
|
Negligible
|
Moderate
|
|
Proxy Perplexity
|
LongLLMLingua
|
Moderate (4x - 10x)
|
Moderate
|
Low to Moderate
|
|
Native Pruning
|
AttentionRAG
|
Moderate (4x - 6x)
|
Varies
|
Very Low
|
|
Trained Synthesis
|
RECOMP, CORE
|
Extreme (10x - 30x)
|
High
|
Low
|
|
KV-Cache Eviction
|
H2O, SnapKV, RazorAttention
|
N/A (decoding-time)
|
Low
|
Very Low
|
Compare KV-cache eviction with heuristic pruning, proxy-perplexity compression, native pruning, and trained synthesis by overhead, semantic-loss risk, and compression behavior. section 13.7
