liveBook · Manning

Native pruning

As explained in Retrieval Augmented Generation, The Foundational Ideas

Native pruning is token or context pruning that derives importance scores directly from a model's internal attention weights.

A compressed prompt can lose the exact passage needed to answer a question. A separate perplexity scorer may discard a predictable contract clause even when the user asks about it. Native pruning instead uses a transformer's attention weights to identify context worth keeping, bringing the pruning signal closer to what a model attends to when forming an answer. section 13.1.1 section 13.3

In AttentionRAG, retrieval first supplies chunks for the user's query. A lightweight model turns the query into an answer-hint prefix with a blank to predict. An open-weight pruning model processes each chunk with that prefix; attention involving the chunk tokens and the blank token is summed across transformer layers, and the highest-scoring sentences form the compressed context. The final answer can still come from a closed-source generator. section 13.3

Figure 13.2 The two obstacles to reading attention, and the answer hint prefix that resolves them. Long sequences dilute softmax mass; multi-token queries scatter focal points. Reformulating the query as a fill-in-the-blank prefix forces a single focal token whose attention concentrates on the context that informs the prediction.

The answer-hint prefix gives attention a single focal token: ordinary long contexts dilute attention, while multi-token questions scatter it. section 13.3

Figure 13.3 The AttentionRAG pipeline. Stage 1 retrieves chunks for the user's query. Stage 2 builds the answer hint prefix from the same query using a lightweight LLM. Stage 3 feeds each chunk and the prefix to the small open-weight pruning model and reads the attention from the chunk's tokens to the prefix's blank token, summed across all transformer layers; the highest-scoring sentences become the compressed context. The final generator can be any model, including a closed-source API.

The pipeline diagram separates the model whose attention guides pruning from the model that generates the final answer. section 13.3

The distinction from LongLLMLingua is the scoring signal. LongLLMLingua uses question-conditioned perplexity from a small proxy; AttentionRAG reads attention from an open-weight model instead. Choose the simpler perplexity approach when deployment ease matters most. AttentionRAG requires access to a model's internal attention, but is aimed at accuracy-sensitive tasks such as multi-hop questions and finding precise answers in long documents. Its compression model need not be the final generator, so a remaining difference between those models should not be mistaken for perfect alignment. section 13.1.1 section 13.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains an answer-hint prefix, layer-summed attention scoring, and sentence selection; does not specify sentence-score aggregation.
Examples
Uses AttentionRAG's retrieved-chunk pipeline and reports TriviaQA exact match of 0.84 at 3.2× compression.
What it assumes
Requires an open-weight model whose internal attention can be read for compression, but not for final generation.
What you can do after
Evaluate/understand it when choosing between attention-based pruning for accuracy-sensitive questions and simpler perplexity-based compression.
How it connects
Connects retrieval, context compression, transformer attention, perplexity scoring, and question answering.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.