liveBook · Manning

Key-value cache

As explained in Retrieval Augmented Generation, The Foundational Ideas

A decoder cache stores key and value attention states from previously processed tokens for reuse during autoregressive generation.

A key-value (KV) cache stores past attention activations while a decoder generates tokens. Reusing these states avoids reprocessing the full prompt at every decoding step, reducing memory or latency costs during long generations. section 13.0 section 13.8.2

KV-cache optimization complements prompt compression rather than replacing it. Prompt compression reduces prefill work before generation; cache eviction reduces decoding memory after generation begins, making it especially useful for long outputs. section 13.8.2 section 13.9

Common strategies trade cache size against retained attention information:

  • H2O maintains a fixed-size cache by evicting tokens with low accumulated attention scores. section 13.8.2
  • SnapKV performs one-shot pruning after prefill, retaining high-attention tokens before decoding. section 13.8.2
  • RazorAttention preserves the full cache for retrieval heads, prunes non-retrieval heads, and uses a compensation token to limit accuracy loss. section 13.8.2
  • ManifoldKV scores tokens by L2 distance from a local-context centroid, preserving salient radial outliers such as names, dates, and numerical values. section 13.8.2

Cached summary states for RAPTOR

A recursive retrieval tree can precompute KV states for its static summary nodes. Retrieval then loads those compressed states instead of reprocessing summary text through the language model for every query, reducing time-to-first-token during deep-tree traversal. section 10.7.2

Table 10.1 RAPTOR failure modes and mitigations
Risk
Description
Mitigation strategy
Tree Construction Cost
Building the tree costs one LLM call per cluster per level. Summarizing a 100k token document involves processing ~150k tokens (input + output) recursively. This scales linearly with document length.
Use cheaper models: Use gpt-4o-mini or claude-3-haiku for the intermediate summarization steps (Claude Haiku, GPT-4o-mini, or similar small models). Save the larger model for the root summary and final generation.
Latency
"Tree Traversal" retrieval (searching layer by layer) adds multiple sequential steps, increasing latency.
Collapsed Tree for sub-million indices (one O(1) vector search). For traversal at scale, precompute and cache the KV states for the upper-layer summary nodes; see KV-RAPTOR below.
Information Loss
A Level 2 summary might abstract away an important detail (e.g., a specific date) found in Level 0. If the retriever only pulls the summary, the fact is lost.
Hybrid Retrieval: Always index the leaf nodes alongside the summaries. Pair with iterative drill-down techniques like FLARE to fall through to leaves if the summary is insufficient.
Cluster Incoherence
GMM might group unrelated chunks if the embeddings are noisy, leading to "hallucinated" summaries that try to bridge unrelated topics.
Tune UMAP: Adjust n_neighbors to enforce tighter local structure. Increase the GMM probability threshold to prevent weak associations.

Use the failure-mode table to compare cache-related latency mitigation with other RAPTOR tradeoffs, including collapsed trees and upper-layer KV-state caching. section 10.7.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains cache eviction strategies and their tradeoffs, but stops short of implementation code or decoder internals.
Examples
Uses H2O, SnapKV, RazorAttention, ManifoldKV, and precomputed RAPTOR summary-node states as concrete examples.
What it assumes
Assumes familiarity with language-model generation and attention, while defining the cache's role from its stored activations.
What you can do after
Operate/configure cache-reduction methods while weighing decoding memory, latency, retained information, and accuracy loss.
How it connects
Connects KV-cache eviction to prompt compression, autoregressive decoding, attention heads, FlashAttention, and retrieval-augmented generation.

Also explained in

Compare how 9 books explain key-value cache →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.