Retrieved passages can contain the answer yet still make a long prompt harder to use. AttentionRAG compresses them before generation by keeping sentences that receive strong attention from a model predicting an answer to the user’s question. Unlike perplexity-based pruning, it scores relevance using transformer attention rather than token predictability. section 13.3 section 13.1.1
The query-time pipeline has three steps:
- Retrieve chunks, then turn the question into an answer-hint prefix with a blank to predict. The single prediction point gives attention a clearer focus than a multi-token question. section 13.3
- Run each chunk and prefix through a small open-weight model. Read attention linking the blank token to chunk tokens and combine scores across transformer layers. section 13.3
- Keep the highest-scoring sentences as compressed context for the final generator, which may be a closed-source API. section 13.3
Figure 13.3 The AttentionRAG pipeline. Stage 1 retrieves chunks for the user's query. Stage 2 builds the answer hint prefix from the same query using a lightweight LLM. Stage 3 feeds each chunk and the prefix to the small open-weight pruning model and reads the attention from the chunk's tokens to the prefix's blank token, summed across all transformer layers; the highest-scoring sentences become the compressed context. The final generator can be any model, including a closed-source API.

Follow the retrieved chunks through prefix construction, attention scoring, and sentence selection; the pruning model and final generator need not be the same model. section 13.3
Reported compression results
With Llama-3.1-Instruct as both compression model and generator, TriviaQA reached 0.84 exact match at 3.2× compression. On 2WikiMQA, the reported ratios were 6.3× with Llama-3.1-Instruct and 15× with Qwen-2.5-Instruct; the table does not give exact-match scores for those rows. These figures are dataset-and-model results, not a safe compression target for every prompt. section 13.3
Choose AttentionRAG when precise long-document or multi-hop answers justify running a model whose attention weights are accessible. LongLLMLingua is simpler to insert before a closed-source generator. AttentionRAG still needs an open-weight model for pruning, even if generation uses an API. Test answer quality as compression rises: removing distractors can help, but excessive pruning can remove meaning. section 13.3 section 13.9
