liveBook · Manning

AttentionRAG

As explained in Retrieval Augmented Generation, The Foundational Ideas

AttentionRAG is a retrieval-context compression technique that uses a transformer’s attention to select relevant tokens and sentences before generation.

Retrieved passages can contain the answer yet still make a long prompt harder to use. AttentionRAG compresses them before generation by keeping sentences that receive strong attention from a model predicting an answer to the user’s question. Unlike perplexity-based pruning, it scores relevance using transformer attention rather than token predictability. section 13.3 section 13.1.1

The query-time pipeline has three steps:

  • Retrieve chunks, then turn the question into an answer-hint prefix with a blank to predict. The single prediction point gives attention a clearer focus than a multi-token question. section 13.3
  • Run each chunk and prefix through a small open-weight model. Read attention linking the blank token to chunk tokens and combine scores across transformer layers. section 13.3
  • Keep the highest-scoring sentences as compressed context for the final generator, which may be a closed-source API. section 13.3
Figure 13.3 The AttentionRAG pipeline. Stage 1 retrieves chunks for the user's query. Stage 2 builds the answer hint prefix from the same query using a lightweight LLM. Stage 3 feeds each chunk and the prefix to the small open-weight pruning model and reads the attention from the chunk's tokens to the prefix's blank token, summed across all transformer layers; the highest-scoring sentences become the compressed context. The final generator can be any model, including a closed-source API.

Follow the retrieved chunks through prefix construction, attention scoring, and sentence selection; the pruning model and final generator need not be the same model. section 13.3

Reported compression results

With Llama-3.1-Instruct as both compression model and generator, TriviaQA reached 0.84 exact match at 3.2× compression. On 2WikiMQA, the reported ratios were 6.3× with Llama-3.1-Instruct and 15× with Qwen-2.5-Instruct; the table does not give exact-match scores for those rows. These figures are dataset-and-model results, not a safe compression target for every prompt. section 13.3

Choose AttentionRAG when precise long-document or multi-hop answers justify running a model whose attention weights are accessible. LongLLMLingua is simpler to insert before a closed-source generator. AttentionRAG still needs an open-weight model for pruning, even if generation uses an API. Test answer quality as compression rises: removing distractors can help, but excessive pruning can remove meaning. section 13.3 section 13.9

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the answer-hint prefix, transformer-attention scoring, and sentence selection, but does not provide implementation code in the supplied evidence.
Examples
Reports TriviaQA exact match at 3.2× compression and compares compression ratios on 2WikiMQA across two model families.
What it assumes
Assumes familiarity with retrieved chunks and transformer models; explains how an answer-hint prefix makes their attention useful for pruning.
What you can do after
Evaluate/understand when attention-guided pruning warrants its open-weight requirement and test answer quality against compression rate.
How it connects
Connects retrieval-augmented generation with context compression, transformer attention, perplexity-based pruning, and extractive sentence selection.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.