A compressed prompt can lose the exact passage needed to answer a question. A separate perplexity scorer may discard a predictable contract clause even when the user asks about it. Native pruning instead uses a transformer's attention weights to identify context worth keeping, bringing the pruning signal closer to what a model attends to when forming an answer. section 13.1.1 section 13.3
In AttentionRAG, retrieval first supplies chunks for the user's query. A lightweight model turns the query into an answer-hint prefix with a blank to predict. An open-weight pruning model processes each chunk with that prefix; attention involving the chunk tokens and the blank token is summed across transformer layers, and the highest-scoring sentences form the compressed context. The final answer can still come from a closed-source generator. section 13.3
Figure 13.2 The two obstacles to reading attention, and the answer hint prefix that resolves them. Long sequences dilute softmax mass; multi-token queries scatter focal points. Reformulating the query as a fill-in-the-blank prefix forces a single focal token whose attention concentrates on the context that informs the prediction.

The answer-hint prefix gives attention a single focal token: ordinary long contexts dilute attention, while multi-token questions scatter it. section 13.3
Figure 13.3 The AttentionRAG pipeline. Stage 1 retrieves chunks for the user's query. Stage 2 builds the answer hint prefix from the same query using a lightweight LLM. Stage 3 feeds each chunk and the prefix to the small open-weight pruning model and reads the attention from the chunk's tokens to the prefix's blank token, summed across all transformer layers; the highest-scoring sentences become the compressed context. The final generator can be any model, including a closed-source API.

The pipeline diagram separates the model whose attention guides pruning from the model that generates the final answer. section 13.3
The distinction from LongLLMLingua is the scoring signal. LongLLMLingua uses question-conditioned perplexity from a small proxy; AttentionRAG reads attention from an open-weight model instead. Choose the simpler perplexity approach when deployment ease matters most. AttentionRAG requires access to a model's internal attention, but is aimed at accuracy-sensitive tasks such as multi-hop questions and finding precise answers in long documents. Its compression model need not be the final generator, so a remaining difference between those models should not be mistaken for perfect alignment. section 13.1.1 section 13.3
