A key-value (KV) cache stores past attention activations while a decoder generates tokens. Reusing these states avoids reprocessing the full prompt at every decoding step, reducing memory or latency costs during long generations. section 13.0 section 13.8.2
KV-cache optimization complements prompt compression rather than replacing it. Prompt compression reduces prefill work before generation; cache eviction reduces decoding memory after generation begins, making it especially useful for long outputs. section 13.8.2 section 13.9
Common strategies trade cache size against retained attention information:
- H2O maintains a fixed-size cache by evicting tokens with low accumulated attention scores. section 13.8.2
- SnapKV performs one-shot pruning after prefill, retaining high-attention tokens before decoding. section 13.8.2
- RazorAttention preserves the full cache for retrieval heads, prunes non-retrieval heads, and uses a compensation token to limit accuracy loss. section 13.8.2
- ManifoldKV scores tokens by L2 distance from a local-context centroid, preserving salient radial outliers such as names, dates, and numerical values. section 13.8.2
Cached summary states for RAPTOR
A recursive retrieval tree can precompute KV states for its static summary nodes. Retrieval then loads those compressed states instead of reprocessing summary text through the language model for every query, reducing time-to-first-token during deep-tree traversal. section 10.7.2
Table 10.1 RAPTOR failure modes and mitigations
|
Risk
|
Description
|
Mitigation strategy
|
|
Tree Construction Cost
|
Building the tree costs one LLM call per cluster per
level. Summarizing a 100k token document involves processing ~150k tokens
(input + output) recursively. This scales linearly with document length.
|
Use cheaper models: Use
gpt-4o-mini or claude-3-haiku for the intermediate summarization steps
(Claude Haiku, GPT-4o-mini, or similar small models). Save the larger model
for the root summary and final generation.
|
|
Latency
|
"Tree Traversal" retrieval (searching layer by
layer) adds multiple sequential steps, increasing latency.
|
Collapsed Tree for sub-million
indices (one O(1) vector search). For traversal at scale, precompute
and cache the KV states for the upper-layer summary nodes; see
KV-RAPTOR below.
|
|
Information Loss
|
A Level 2 summary might abstract away an important detail
(e.g., a specific date) found in Level 0. If the retriever only pulls the
summary, the fact is lost.
|
Hybrid Retrieval: Always index the
leaf nodes alongside the summaries. Pair with iterative drill-down techniques
like FLARE to fall through to leaves if the summary
is insufficient.
|
|
Cluster Incoherence
|
GMM might group unrelated chunks if the embeddings are
noisy, leading to "hallucinated" summaries that try to bridge
unrelated topics.
|
Tune UMAP: Adjust n_neighbors to enforce tighter
local structure. Increase the GMM probability threshold to prevent weak
associations.
|
Use the failure-mode table to compare cache-related latency mitigation with other RAPTOR tradeoffs, including collapsed trees and upper-layer KV-state caching. section 10.7.2








