liveBook · Manning

Kv-cache eviction

As explained in Retrieval Augmented Generation, The Foundational Ideas

Key-value cache eviction is the removal of stored transformer attention states during decoding to control memory use.

KV-cache eviction controls decoding memory by removing stored key-value attention states while leaving the input prompt unchanged. It complements prompt compression: compression reduces prefill cost, whereas eviction reduces memory during autoregressive generation. section 13.8.2 section 13.0

Different eviction strategies make different compromises between simplicity, selectivity, and accuracy:

  • H2O maintains a fixed-size cache and evicts tokens with low accumulated attention scores. section 13.8.2
  • SnapKV performs one-shot pruning after prefill by retaining high-attention hit tokens before generation. section 13.8.2
  • RazorAttention preserves full caches for retrieval heads, prunes non-retrieval heads, and uses a compensation token; it reduces cache size by roughly 70 percent without retraining. section 13.8.2
  • ManifoldKV scores tokens by L2 distance from a local context centroid, preserving salient radial outliers such as names, dates, and numerical values. section 13.8.2

Needle retrieval under compression

On a three-key needle-in-a-haystack task at 50 percent compression, ManifoldKV outperformed cosine-based alternatives by 15.4 points. The reported geometric advantage remains promising rather than settled because the work is a February 2026 preprint and has not been peer-reviewed. section 13.8.2

Table 13.4 Method tradeoffs
Compression paradigm
Primary methodology
Optimal ratio
Compute overhead
Risk of semantic loss
Heuristic Pruning
TF-IDF Filtering
Low (~2x)
Negligible
Moderate
Proxy Perplexity
LongLLMLingua
Moderate (4x - 10x)
Moderate
Low to Moderate
Native Pruning
AttentionRAG
Moderate (4x - 6x)
Varies
Very Low
Trained Synthesis
RECOMP, CORE
Extreme (10x - 30x)
High
Low
KV-Cache Eviction
H2O, SnapKV, RazorAttention
N/A (decoding-time)
Low
Very Low

Compare KV-cache eviction with heuristic pruning, proxy-perplexity compression, native pruning, and trained synthesis by overhead, semantic-loss risk, and compression behavior. section 13.7

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains cache construction, eviction timing, attention-head selectivity, geometric scoring, and reported tradeoffs without providing implementation code.
Examples
Uses H2O, SnapKV, RazorAttention, ManifoldKV, and a three-key needle-in-a-haystack evaluation.
What you can do after
Evaluate eviction strategies against prompt compression by matching cache reduction, latency constraints, accuracy risk, and workload shape.
How it connects
Connects prompt compression, autoregressive decoding, attention heads, retrieval heads, FlashAttention, and semantic loss.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.