A retrieved document can contain the answer while a generator still misses it. Long contexts bury relevant facts among distracting passages, increase cost, and slow processing. Pruning removes lower-value passages or tokens before generation so the remaining context is easier to use. section 13.0 section 5.6.1
The choice is what to remove and how to judge its value:
- Passage pruning keeps a smaller candidate set. KG-FiD links related passages in a graph, refines their embeddings with a graph neural network, then reranks and keeps selected passages for the FiD encoder and decoder. section 5.6.1
- Token pruning removes expendable pieces within retrieved text. A proxy language model can drop predictable filler, but predictability alone ignores the question. LongLLMLingua compares token scores with and without the question to make pruning query-aware. section 13.0 section 13.1.1
- AttentionRAG instead uses the generator’s own attention weights to score context. This avoids disagreement between a proxy scorer and the model that must answer, but requires access to the generator’s attention. section 13.1.1
When a small deletion changes the answer
A standard termination clause may look predictable to a proxy model yet be exactly what a user requested. At higher compression, dropping a modifier such as “Except” can reverse a clause. Keep critical terms explicitly rather than assuming a good overall compression score protects them. section 13.1.1 section 13.7
Table 13.4 Method tradeoffs
|
Compression paradigm
|
Primary methodology
|
Optimal ratio
|
Compute overhead
|
Risk of semantic loss
|
|
Heuristic Pruning
|
TF-IDF Filtering
|
Low (~2x)
|
Negligible
|
Moderate
|
|
Proxy Perplexity
|
LongLLMLingua
|
Moderate (4x - 10x)
|
Moderate
|
Low to Moderate
|
|
Native Pruning
|
AttentionRAG
|
Moderate (4x - 6x)
|
Varies
|
Very Low
|
|
Trained Synthesis
|
RECOMP, CORE
|
Extreme (10x - 30x)
|
High
|
Low
|
|
KV-Cache Eviction
|
H2O, SnapKV, RazorAttention
|
N/A (decoding-time)
|
Low
|
Very Low
|
Compare heuristic, proxy-scored, and generator-native pruning against trained synthesis and decoding-time cache eviction. The table separates their typical compression ranges, compute costs, and semantic-loss risks. section 13.7
Pruning has a semantic limit: aggressive deletion can break entities, dates, or relationships between facts. For factual recall where the answer is a span of original text, test an extractive approach first. For summaries or multi-hop answers requiring facts from different documents, consider synthesis instead. Start around 4x compression; compare answers against an uncompressed pipeline on held-out queries, increase the ratio only while accuracy holds, and force-keep terms whose loss would change meaning. KV-cache eviction is a separate, decoding-time way to reduce memory without changing the prompt. section 13.4.1 section 13.7 section 13.0










