liveBook · Manning

Pruning

As explained in Retrieval Augmented Generation, The Foundational Ideas

Pruning is the removal of less relevant branches or items from a candidate set during retrieval.

A retrieved document can contain the answer while a generator still misses it. Long contexts bury relevant facts among distracting passages, increase cost, and slow processing. Pruning removes lower-value passages or tokens before generation so the remaining context is easier to use. section 13.0 section 5.6.1

The choice is what to remove and how to judge its value:

  • Passage pruning keeps a smaller candidate set. KG-FiD links related passages in a graph, refines their embeddings with a graph neural network, then reranks and keeps selected passages for the FiD encoder and decoder. section 5.6.1
  • Token pruning removes expendable pieces within retrieved text. A proxy language model can drop predictable filler, but predictability alone ignores the question. LongLLMLingua compares token scores with and without the question to make pruning query-aware. section 13.0 section 13.1.1
  • AttentionRAG instead uses the generator’s own attention weights to score context. This avoids disagreement between a proxy scorer and the model that must answer, but requires access to the generator’s attention. section 13.1.1

When a small deletion changes the answer

A standard termination clause may look predictable to a proxy model yet be exactly what a user requested. At higher compression, dropping a modifier such as “Except” can reverse a clause. Keep critical terms explicitly rather than assuming a good overall compression score protects them. section 13.1.1 section 13.7

Table 13.4 Method tradeoffs
Compression paradigm
Primary methodology
Optimal ratio
Compute overhead
Risk of semantic loss
Heuristic Pruning
TF-IDF Filtering
Low (~2x)
Negligible
Moderate
Proxy Perplexity
LongLLMLingua
Moderate (4x - 10x)
Moderate
Low to Moderate
Native Pruning
AttentionRAG
Moderate (4x - 6x)
Varies
Very Low
Trained Synthesis
RECOMP, CORE
Extreme (10x - 30x)
High
Low
KV-Cache Eviction
H2O, SnapKV, RazorAttention
N/A (decoding-time)
Low
Very Low

Compare heuristic, proxy-scored, and generator-native pruning against trained synthesis and decoding-time cache eviction. The table separates their typical compression ranges, compute costs, and semantic-loss risks. section 13.7

Pruning has a semantic limit: aggressive deletion can break entities, dates, or relationships between facts. For factual recall where the answer is a span of original text, test an extractive approach first. For summaries or multi-hop answers requiring facts from different documents, consider synthesis instead. Start around 4x compression; compare answers against an uncompressed pipeline on held-out queries, increase the ratio only while accuracy holds, and force-keep terms whose loss would change meaning. KV-cache eviction is a separate, decoding-time way to reduce memory without changing the prompt. section 13.4.1 section 13.7 section 13.0

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains passage selection and token scoring, then gives compression-ratio checks and semantic-loss safeguards.
Examples
Uses a termination-clause query, a dropped “Except” modifier, and KG-FiD passage reranking to expose different pruning decisions.
What it assumes
Introduces the retrieved-answer failure and explains proxy scoring before contrasting it with generator-native attention.
What you can do after
Apply it via LongLLMLingua or AttentionRAG, validating against uncompressed answers and protecting terms that can reverse meaning.
How it connects
Connects pruning to retrieval-augmented generation, context compression, perplexity, attention weights, graph neural networks, and abstractive synthesis.

Also explained in

Compare how 11 books explain pruning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.