Chapter 13: Context compression and pruning
This introduction explains why long retrieved context can hurt RAG answers and outlines ways to reduce or prioritize what the generator processes.
After this chapter you will be able to
- Long context extraction failures. Explain how a RAG system can retrieve the answer yet fail to use it, and why reducing context can improve both efficiency and answer quality.
- Evidence prioritization principles. Describe how the query, retrieval confidence, prompt position, and available generator attention signals can guide what context to keep or emphasize.
- Contextual dilution and position. Explain why concatenating retrieved documents can cause a model to miss relevant evidence, especially when it is buried among other text.
- Token costs and latency. Explain how longer retrieved contexts raise inference expense and slow the start of an answer.
- Compression before the generator. Describe where context compression acts in the RAG pipeline and what it changes.
- Limits of query-agnostic scoring. Explain how lexical heuristics and absolute perplexity decide what to discard, and why either can remove the answer to a user's question.
- Question-conditioned perplexity scoring. Describe how LongLLMLingua makes proxy-model pruning query-aware by comparing token predictability with and without the user's question.
- Generator-native relevance scoring. Explain why AttentionRAG scores context using the generator's own attention rather than a proxy model.
- Limits of token pruning. Explain why trained extractive and abstractive compressors can compress beyond the point where token-level pruning breaks meaning, distinguishing RECOMP from CORE.
- Compression scope and fit. Distinguish token-, sentence-, and document-level compression by what each removes, and explain when each is useful or risky for retrieved context.
- Generator access modes. Distinguish black-box from white-box compression by the generator access each requires and the practical tradeoff.
- Proxy-perplexity practical appeal. Explain why proxy-perplexity compression is presented as a practical choice for teams and what LongLLMLingua demonstrated.
- Perplexity as token surprise. Describe how the section relates a token's probability to perplexity and interpret low versus high perplexity using its examples.
- Question-blind perplexity failure. Explain why absolute perplexity can filter out a relevant retrieved chunk while retaining high-entropy noise, and contrast that with selecting sentences for their usefulness to the user's question.
- Contrastive perplexity relevance signal. Explain how LongLLMLingua compares a token’s perplexity with and without the user question, and why a large drop signals relevance.
- Anchored document filtering. Explain how the budget controller uses a proxy model to score whole retrieved documents against the user's question, and why an anchor sentence helps prevent misleading scores.
- Query-aware segmented token pruning. Explain how contrastive perplexity guides token pruning and why the pipeline processes configurable segments using preceding compressed segments as context.
- Output fragment recovery. Explain why pruning can leave broken entities in the generated answer and how subsequence recovery repairs them.
- Proxy and generator roles. Explain how LongLLMLingua separates compression from answer generation and when that split is worth using.
- LongLLMLingua configuration choices. Describe how to select the question-aware LongLLMLingua path and what its main compression settings do.
- Legal answer preservation. Trace why the compressed legal context still supports the original answer despite substantial pruning.
- Question-blind compression failure. Distinguish LLMLingua-2 from LongLLMLingua by explaining the legal example's reversed answer and the limits of token protections.
- Operational compression limits. Explain the main costs and failure boundaries to weigh before deploying LongLLMLingua.
- Attention versus perplexity signals. Distinguish AttentionRAG’s attention-based relevance signal from LongLLMLingua’s perplexity-based signal, and explain why the difference matters when compression is aggressive.
- Answer hint attention anchor. Explain the two obstacles to reading useful attention from a long context and multi-token question, and how an answer hint prefix creates a focal prediction.
- Chunk scoring and sentence selection. Trace how AttentionRAG scores retrieved context in chunks and turns those scores into a compressed prompt.
- Evaluator head token pruning. Trace how EHPC uses selected attention heads in early layers to score token importance, drops low-value tokens, and continues the forward pass with fewer tokens in the deeper layers.
- AttentionRAG and EHPC tradeoff. Distinguish AttentionRAG's robust all-layer aggregation from EHPC's faster evaluator-head shortcut, and describe when the section recommends each.
- Task-aligned compression. Explain why RECOMP trains dedicated compressors offline using downstream exact match and F1 signals, rather than relying on inference-time token pruning alone.
- Extractive and abstractive variants. Distinguish RECOMP's extractive sentence selection from its abstractive rewriting, and explain why combining evidence across documents can call for synthesis.
- Selective augmentation and refusal. Describe what RECOMP returns when retrieval contains no useful information and how that supports an explicit refusal path.
- CORE task-oriented training. Explain how CORE avoids human-annotated compression labels while training a compressor for the downstream answering task.
- Multi-hop summary synthesis. Explain why standard token pruning is inadequate for multi-hop tasks and what BRIEF-Pro does instead.
- CASC intermediate synthesis. Trace how CASC prepares retrieved scientific material before the main generator answers.
- Compression format and hallucination risk. Distinguish when the section favors extractive versus abstractive compression and identify the risk introduced by abstraction.
- Retrieve Compress Generate Flow. Explain how adding a query-aware compressor changes a direct retriever-to-generator flow and why orchestration is needed.
- Postprocessor Integration and Budget. Describe how the LlamaIndex and llmlingua wrapper fits into the query pipeline, what target_token controls, and what remains unchanged for callers.
- Recursive retrieval through index nodes. Trace how LlamaIndex uses an IndexNode summary and its index_id link to retrieve original detail on demand, from ingestion through query-time retrieval.
- Recovering overlooked legal clauses. Explain how compression helped the assistant use an indemnification clause that retrieval had already found but the generator overlooked.
- Risks of over-pruning evidence. Describe how stronger compression damaged the legal answer and how the team responded.
- Compression economics and latency. Justify why compression was useful for Meridian despite its weak cost savings, and explain how query volume changes the cost argument.
- Compression and answer quality. Explain how answer quality changes as compression increases, including why moderate compression can help and why excessive pruning can reverse meaning.
- Validated compression ratio selection. Describe how to test and adjust a compression ratio and protect terms whose removal would change the answer.
- Gist Prefix Compression. Explain how training folds a long system prompt into a short gist prefix that replaces the prompt at inference time.
- Gisting Tradeoffs and Fit. Justify when gisting is attractive and why it cannot be applied to a frozen API model.
- Complementary compression costs. Distinguish prompt compression from KV-cache eviction by when each operates and which cost it reduces. Describe which workload favors each approach.
- Visual context compression. Explain why multi-modal RAG needs compression beyond the text-only methods discussed earlier, and describe how visual document representations and hybrid systems can fit text and images into context.
- Confidence-based generation budget. Explain how Adaptive Budget Forcing uses token entropy as a confidence proxy to decide when to stop generation early, instead of relying on a fixed budget.
- Hardware-specific latency target. Describe how HAP-E uses a latency predictor to meet a millisecond-per-token target on specific hardware, and why this differs from optimizing a compression ratio.
- Agentic self-compression tradeoff. Explain how an agent can decide to prune its context during reasoning instead of relying on one upfront compression step, and why that helps multi-hop tasks but adds latency.
What it builds on
- The challenges of multiple document contexts (chapter 5) · The challenge of giving a generator multiple document contexts.
- The RAG model architecture (chapter 4) · The standard RAG pipeline that puts retrieved documents in the generator's prompt.
- Generating hypothetical documents (chapter 7) · The idea of turning a question into answer-shaped text, used here for pruning rather than retrieval.
- From lists to trees (chapter 10) · Introduces organizing retrieved material beyond a flat list.
- The core architecture of graph-based RAG (chapter 12) · Introduces graph-based retrieval architectures.
Section by section
Introduction
This introduction explains why long retrieved context can hurt RAG answers and outlines ways to reduce or prioritize what the generator processes.
- Long context extraction failures: Explain how a RAG system can retrieve the answer yet fail to use it, and why reducing context can improve both efficiency and answer quality.
Why it matters: Retrieving the right document is not enough if the generator misses its answer amid other text. - Context reduction approaches: Distinguish token-level pruning, trained abstractive compression, and KV-cache eviction by what each reduces.
Why it matters: These approaches save resources at different points, so they should not be treated as interchangeable prompt edits. - Evidence prioritization principles: Describe how the query, retrieval confidence, prompt position, and available generator attention signals can guide what context to keep or emphasize.
Why it matters: Compression is most useful when it preserves the evidence this generator needs for this question.
Key terms: extraction failure, lost in the middle, context compression, signal-to-noise ratio, token-level pruning, trained abstractive compressors, KV-cache eviction, query, token budget, positional bias
13.1 The costs of long context
Explains how long retrieved contexts can weaken answer extraction and increase cost and latency, and why removing irrelevant tokens before generation helps.
- Contextual dilution and position: Explain why concatenating retrieved documents can cause a model to miss relevant evidence, especially when it is buried among other text.
Why it matters: Retrieving the right passage does not guarantee that the generator will use it. - Long spans without word matches: Distinguish the long-context difficulty shown by NoLiMa from the easier condition in needle-in-a-haystack tests.
Why it matters: A model's advertised context window or strong literal-match test score may overstate how well it handles ordinary retrieved passages. - Token costs and latency: Explain how longer retrieved contexts raise inference expense and slow the start of an answer.
Why it matters: Context length is an operating-cost and responsiveness decision, not just an answer-quality decision. - Compression before the generator: Describe where context compression acts in the RAG pipeline and what it changes.
Why it matters: Compression can reduce input-token spending while making retrieved evidence less cluttered for an unchanged generator.
Key terms: contextual dilution, self-attention, primacy bias, recency bias, parametric priors, NoLiMa, literal overlap, needle-in-a-haystack tests, input tokens, time-to-first-token
13.1.1 Evolution of context compression
This section traces context compression as a series of fixes for methods that discard useful answer evidence or misjudge what the generator needs.
- Limits of query-agnostic scoring: Explain how lexical heuristics and absolute perplexity decide what to discard, and why either can remove the answer to a user's question.
Why it matters: Cheap or surprising-looking text is not necessarily the text needed to answer the question. - Question-conditioned perplexity scoring: Describe how LongLLMLingua makes proxy-model pruning query-aware by comparing token predictability with and without the user's question.
Why it matters: The comparison helps preserve content tied to the actual question rather than merely unusual content. - Generator-native relevance scoring: Explain why AttentionRAG scores context using the generator's own attention rather than a proxy model.
Why it matters: A compressed prompt is useful only if the answering model can use what remains. - Limits of token pruning: Explain why trained extractive and abstractive compressors can compress beyond the point where token-level pruning breaks meaning, distinguishing RECOMP from CORE.
Why it matters: When further deletion would destroy the evidence, selecting or rewriting coherent text offers another route to a shorter context.
Key terms: lexical heuristics, TF-IDF, BM25, proxy-model perplexity, absolute perplexity, query-agnostic, contrastive perplexity, LongLLMLingua, query-aware, proxy
13.1.2 Types of compression
This section classifies context-compression methods by what they remove and whether they need access to the generator’s internals.
- Compression scope and fit: Distinguish token-, sentence-, and document-level compression by what each removes, and explain when each is useful or risky for retrieved context.
Why it matters: Choosing the wrong scope can discard needed evidence or leave too much irrelevant text. - Generator access modes: Distinguish black-box from white-box compression by the generator access each requires and the practical tradeoff.
Why it matters: The generator you can access determines which compression signals are available and how much infrastructure you must manage.
Key terms: scope, token-level methods, sentence-level methods, document-level methods, access mode, black-box methods, white-box methods, closed API, open-weight LLM, attention weights
13.2 Perplexity-based compression with LongLLMLingua
The section explains why proxy-perplexity compression is practical and introduces the perplexity signal that LongLLMLingua conditions on the user's question.
- Proxy-perplexity practical appeal: Explain why proxy-perplexity compression is presented as a practical choice for teams and what LongLLMLingua demonstrated.
Why it matters: This explains why prompt compression can be useful even when a team cannot change its main model. - Perplexity as token surprise: Describe how the section relates a token's probability to perplexity and interpret low versus high perplexity using its examples.
Why it matters: Perplexity gives meaning to the scores discussed in the compression approach. - Question-conditioned perplexity signal: Distinguish LongLLMLingua's question-conditioned perplexity score from reading perplexity in isolation.
Why it matters: The distinction connects the compression signal to the question the prompt is meant to answer.
Key terms: proxy-perplexity, LongLLMLingua, open-weight scorer, perplexity, token, low perplexity, high perplexity, user's question
13.2.1 Why absolute perplexity fails
This subsection explains why question-blind perplexity can remove useful retrieved text and contrasts it with question-aware selection.
- Question-blind perplexity failure: Explain why absolute perplexity can filter out a relevant retrieved chunk while retaining high-entropy noise, and contrast that with selecting sentences for their usefulness to the user's question.
Why it matters: Compression can defeat retrieval if it discards evidence merely because that evidence sounds ordinary.
Key terms: absolute perplexity, contrastive perplexity, high-entropy noise
13.2.2 Contrastive perplexity as a relevance signal
This section explains how LongLLMLingua uses a question-dependent change in token perplexity to decide which document tokens to keep.
- Contrastive perplexity relevance signal: Explain how LongLLMLingua compares a token’s perplexity with and without the user question, and why a large drop signals relevance.
Why it matters: The score distinguishes information useful for this question from tokens that are merely surprising in general. - Question-specific token selection: Apply the contrastive score to the aspirin-dose example, distinguishing the answer token from a common word and a surprising but off-topic medical term.
Why it matters: The example shows why neither commonness nor unusualness alone determines whether a token helps answer the question.
Key terms: conditional perplexity, contrastive perplexity, user question, relevance signal, proxy model, 81 mg, contrastive score, thrombocytopenia, ground-truth answer
13.2.3 The LongLLMLingua pipeline
This section explains how LongLLMLingua filters, reorders, and prunes retrieved context, then repairs fragments in the generated answer.
- Anchored document filtering: Explain how the budget controller uses a proxy model to score whole retrieved documents against the user's question, and why an anchor sentence helps prevent misleading scores.
Why it matters: Filtering whole documents first saves fine-grained work on context unlikely to help answer the question. - Boundary-focused document ordering: Describe how surviving documents are placed in the final prompt and why the highest-ranked ones go at its boundaries.
Why it matters: Ordering affects whether the generator attends to the most relevant surviving documents. - Query-aware segmented token pruning: Explain how contrastive perplexity guides token pruning and why the pipeline processes configurable segments using preceding compressed segments as context.
Why it matters: Segmentation helps token pruning reflect the question rather than where a token happened to appear in a long document. - Rank-based compression budgets: Describe how the coarse document ranking determines each surviving document's token budget.
Why it matters: The pipeline does not spend the same number of retained tokens on documents with different relevance rankings. - Output fragment recovery: Explain why pruning can leave broken entities in the generated answer and how subsequence recovery repairs them.
Why it matters: Compression should not leave names, dates, or article numbers broken in the answer returned to the user.
Key terms: budget controller, coarse-grained compression, proxy language model, anchor sentence, perplexity, reordering, relevance scores, U-shaped attention curve, contrastive perplexity, query-aware
13.2.4 Implementing a prompt compressor with the LLMLingua
This section shows how to configure and assess LongLLMLingua prompt compression, using a legal-document example to expose both its benefits and its risks.
- Proxy and generator roles: Explain how LongLLMLingua separates compression from answer generation and when that split is worth using.
Why it matters: Compression is not free, so the separation only helps when the saved context is worth the proxy's work. - LongLLMLingua configuration choices: Describe how to select the question-aware LongLLMLingua path and what its main compression settings do.
Why it matters: A single constructor choice can silently switch you away from the question-aware algorithm you meant to run. - Legal answer preservation: Trace why the compressed legal context still supports the original answer despite substantial pruning.
Why it matters: The example shows that preserving the answer-bearing exception matters more than preserving every sentence. - Question-blind compression failure: Distinguish LLMLingua-2 from LongLLMLingua by explaining the legal example's reversed answer and the limits of token protections.
Why it matters: A better compression ratio and polished answer can conceal a reversal of the source's meaning. - Measured compression benefits: State the reported quality, cost, and latency benefits and their stated causes.
Why it matters: These results illustrate why compression may improve more than the token bill, while remaining benchmark-specific. - Operational compression limits: Explain the main costs and failure boundaries to weigh before deploying LongLLMLingua.
Why it matters: Token savings need to be judged alongside compute costs, damaged evidence, and effects on downstream tools.
Key terms: proxy model, target generator, PromptCompressor, effective context window, use_llmlingua2=False, causal language model, condition_compare, reorder_context, dynamic_context_compression_ratio, coarse ranker
13.3 Native pruning with AttentionRAG
This section explains how AttentionRAG uses a small model’s attention weights to prune retrieved context, why it uses an answer-shaped prefix, and what its accuracy and deployment tradeoffs are.
- Attention versus perplexity signals: Distinguish AttentionRAG’s attention-based relevance signal from LongLLMLingua’s perplexity-based signal, and explain why the difference matters when compression is aggressive.
Why it matters: The signal used to decide what to remove can determine whether useful evidence survives compression. - Answer hint attention anchor: Explain the two obstacles to reading useful attention from a long context and multi-token question, and how an answer hint prefix creates a focal prediction.
Why it matters: A single prediction target makes attention more useful for identifying context to preserve. - Chunk scoring and sentence selection: Trace how AttentionRAG scores retrieved context in chunks and turns those scores into a compressed prompt.
Why it matters: This is how attention scores become a shorter prompt without reducing the retained evidence to isolated tokens. - Pruning and generation roles: Describe the separate jobs of AttentionRAG’s pruning model and final generator, including which one requires accessible attention weights.
Why it matters: Separating relevance scoring from answer writing makes the approach modular without requiring access to the final generator’s internals. - Compression performance and constraints: Describe the reported compression–accuracy tradeoff and the deployment limitation created by AttentionRAG’s variable compression ratio.
Why it matters: A useful compression method still has to balance answer quality against prompt size and operational limits.
Key terms: native pruning, attention weights, perplexity, alignment gap, attention dilution, query misalignment, answer hint prefix, chunks, attention features, all-layer aggregation
13.3.1 Beyond all-layer aggregation: Evaluator heads
This subsection explains how EHPC uses selected early attention heads to prune tokens before expensive layers, and when its speed advantage is worth the tradeoff against AttentionRAG.
- Evaluator head token pruning: Trace how EHPC uses selected attention heads in early layers to score token importance, drops low-value tokens, and continues the forward pass with fewer tokens in the deeper layers.
Why it matters: Pruning before the deeper layers reduces the work they do on every query. - AttentionRAG and EHPC tradeoff: Distinguish AttentionRAG's robust all-layer aggregation from EHPC's faster evaluator-head shortcut, and describe when the section recommends each.
Why it matters: The choice depends on whether answer accuracy or inference speed is the tighter constraint.
Key terms: EHPC, evaluator heads, token importance, forward pass, AttentionRAG, all-layer sum, latency, throughput
13.4 Extensions: The RECOMP framework
This section explains how RECOMP goes beyond token pruning by training compressors for answer quality, allowing rewritten evidence, and withholding context when retrieval is unhelpful.
- Task-aligned compression: Explain why RECOMP trains dedicated compressors offline using downstream exact match and F1 signals, rather than relying on inference-time token pruning alone.
Why it matters: A shorter prompt is not necessarily a better prompt for answering the question. - Extractive and abstractive variants: Distinguish RECOMP's extractive sentence selection from its abstractive rewriting, and explain why combining evidence across documents can call for synthesis.
Why it matters: Selecting evidence and expressing a conclusion drawn from several sources solve different compression problems. - Selective augmentation and refusal: Describe what RECOMP returns when retrieval contains no useful information and how that supports an explicit refusal path.
Why it matters: Withholding bad context can be safer than giving the generator a plausible-looking summary of irrelevant evidence. - Benchmark tradeoffs and synthesis risk: Describe how the reported single-document and multi-document results qualify the appeal of abstractive synthesis.
Why it matters: The option that can express a cross-document answer also carries a reliability risk in the reported multi-document benchmark.
Key terms: RECOMP, token pruning, exact match, F1, extractive compressor, abstractive compressor, abstractive synthesis, selective augmentation, empty string, I don't know
13.4.1 CORE and multi-hop synthesis
This section explains how CORE trains task-oriented compression without human labels, why multi-hop tasks need synthesis rather than token pruning, and how to choose between extractive and abstractive compression.
- CORE task-oriented training: Explain how CORE avoids human-annotated compression labels while training a compressor for the downstream answering task.
Why it matters: A compressor can be trained for answer usefulness without continually collecting human compression labels. - GRPO candidate comparison: Describe how CORE’s GRPO compares candidate summaries to produce a training signal without a separate critic model.
Why it matters: This explains how CORE can compare compressions during refinement without a memory-intensive critic. - Multi-hop summary synthesis: Explain why standard token pruning is inadequate for multi-hop tasks and what BRIEF-Pro does instead.
Why it matters: Keeping isolated tokens may lose the connections an answer must draw across sources. - CASC intermediate synthesis: Trace how CASC prepares retrieved scientific material before the main generator answers.
Why it matters: The generator receives a shorter, reconciled account of the retrieved evidence instead of raw passages. - Compression format and hallucination risk: Distinguish when the section favors extractive versus abstractive compression and identify the risk introduced by abstraction.
Why it matters: A shorter context is not useful if its format fails the task or its compressor introduces unsupported facts.
Key terms: CORE, knowledge distillation warm-up, RL policy refinement, exact match, F1, static black-box environment, Group Relative Policy Optimization, GRPO, critic-free, group-normalized empirical advantage
13.5 Implementing a query-aware compressor
The section shows how to insert query-aware compression into a RAG pipeline and explains its effects on the query interface, token budget, and operational logging.
- Retrieve Compress Generate Flow: Explain how adding a query-aware compressor changes a direct retriever-to-generator flow and why orchestration is needed.
Why it matters: The compressor needs a defined place in the pipeline and routing for both the query and retrieved text. - Postprocessor Integration and Budget: Describe how the LlamaIndex and llmlingua wrapper fits into the query pipeline, what target_token controls, and what remains unchanged for callers.
Why it matters: The pipeline can reduce retrieved text without requiring callers to adopt a different query interface. - Compressed Prompt Logging: Explain the section's logging rule for production RAG and how prompt compression affects log volume.
Why it matters: Logging choices can otherwise erase some of compression's operational savings.
Key terms: query-aware compressor, retrieve-compress-generate topology, orchestrator, pruned payload, LlamaIndex, llmlingua, postprocessor, target_token, query engine, uncompressed prompts
13.5.1 Recursive retrieval and index nodes
This section explains how recursive retrieval uses compressed summaries and links to original document chunks to reduce context without losing hierarchical detail.
- Hierarchy loss from chunk pruning: Explain why standard chunk-level compression can be destructive for complex hierarchical documents such as nested markdown or multi-tiered financial reports.
Why it matters: Preserving relevant text is not enough if pruning loses how that text fits into the document. - Recursive retrieval through index nodes: Trace how LlamaIndex uses an IndexNode summary and its index_id link to retrieve original detail on demand, from ingestion through query-time retrieval.
Why it matters: The system can retrieve compact summaries first while retaining access to granular evidence when a query calls for it.
Key terms: chunk-level compression, hierarchical documents, relational meaning, recursive retrieval, LlamaIndex, IndexNode, index_id, compressed summaries, uncompressed child nodes
13.6 Case study: Legal document synthesis
Legal case studies show how compression can improve retrieval-augmented answers, where excessive pruning can damage them, and when latency or scale justifies the work.
- Recovering overlooked legal clauses: Explain how compression helped the assistant use an indemnification clause that retrieval had already found but the generator overlooked.
Why it matters: A retrieval failure and a failure to use retrieved evidence call for different fixes. - Risks of over-pruning evidence: Describe how stronger compression damaged the legal answer and how the team responded.
Why it matters: A shorter prompt is not an improvement if it removes the evidence needed for a faithful, citable answer. - Compression economics and latency: Justify why compression was useful for Meridian despite its weak cost savings, and explain how query volume changes the cost argument.
Why it matters: The reason to build compression depends on workload scale and the value of responsiveness, not token savings alone. - Hierarchical litigation-file summaries: Trace how grounded retrieval and successive summaries reduced a large litigation file to a usable context.
Why it matters: Staged, evidence-grounded synthesis keeps a long timeline manageable without asking the model to read the entire file at once. - Restoring linked contract definitions: Describe how a lexical graph helps preserve the meaning of a compressed contract clause.
Why it matters: A short excerpt may be legally incomplete unless its referenced definitions accompany it.
Key terms: LongLLMLingua, failure point 5, compression ratio, failure point 4, contrastive perplexity, keep_first_sentence, token-level pruning, financial break-even point, time-to-first-token, hierarchical summarization
13.7 Tradeoffs: Compression ratio versus semantic loss
This section explains how to choose and validate a context-compression strategy that saves tokens without losing facts needed for accurate answers.
- Compression and answer quality: Explain how answer quality changes as compression increases, including why moderate compression can help and why excessive pruning can reverse meaning.
Why it matters: The best ratio is not necessarily the highest one: it is the one that preserves answer quality for the task. - Validated compression ratio selection: Describe how to test and adjust a compression ratio and protect terms whose removal would change the answer.
Why it matters: Testing and protecting critical words helps find token savings without silently changing the evidence. - Task matched compression style: Distinguish when to use extractive compression and when to use abstractive synthesis.
Why it matters: The form of the needed answer helps determine what evidence the compressor should preserve. - Access matched compression method: Explain how access to the model affects the choice between proxy-perplexity and native-attention compression.
Why it matters: A method must fit the model access your deployment actually provides. - KV-cache eviction boundary: Distinguish the decoding-memory problem addressed by KV-cache eviction from the input-token problem addressed by prompt compression.
Why it matters: Reducing input tokens alone may not address the resource constraint in a long-running conversation.
Key terms: Pareto frontier, compression ratio, semantic loss, noise removal zone, safe operating range, held-out query set, exact match, force tokens, must-keep strings, domain-critical terminology
13.8 Future directions and adjacent research
Introduces five emerging directions in context compression beyond runtime input-prompt compression.
13.8.1 Gist tokens and trained prompt compression
This subsection explains how trained gist tokens replace a long prompt and when that approach is practical.
- Gist Prefix Compression: Explain how training folds a long system prompt into a short gist prefix that replaces the prompt at inference time.
Why it matters: It distinguishes learned prompt representations from methods that shorten a prompt by removing its original tokens. - Gisting Tradeoffs and Fit: Justify when gisting is attractive and why it cannot be applied to a frozen API model.
Why it matters: The memory benefit is useful only when you can train the model and reuse the compressed prompt enough to justify that training.
Key terms: gist prefix, virtual continuous tokens, key/value activations, gist caching, instruction caching, compress-by-training, runtime pruning
13.8.2 KV-cache eviction
This section explains how KV-cache eviction reduces decoding memory, surveys four eviction approaches, and distinguishes when it is more useful than prompt compression.
- Complementary compression costs: Distinguish prompt compression from KV-cache eviction by when each operates and which cost it reduces. Describe which workload favors each approach.
Why it matters: The bottleneck determines which kind of compression is likely to help most. - Attention based cache pruning: Describe how H2O, SnapKV, and RazorAttention select cache content to retain or evict.
Why it matters: These approaches differ in when they prune and whether they make decisions by token or by attention head. - ManifoldKV geometric scoring: Explain why ManifoldKV uses L2 distance from a local context centroid instead of cosine-based scoring, and qualify the reported geometry finding.
Why it matters: The scoring choice aims to retain important names, dates, and numbers without assuming that the reported cross-model geometry is settled.
Key terms: prompt compression, prefill cost, KV-cache eviction, decoding memory cost, H2O, SnapKV, RazorAttention, retrieval heads, compensation token, ManifoldKV
13.8.3 Multi-modal retrieval
This section explains why retrieval across text and images needs visual-token compression and describes approaches to representing visual documents and selecting informative examples.
- Visual context compression: Explain why multi-modal RAG needs compression beyond the text-only methods discussed earlier, and describe how visual document representations and hybrid systems can fit text and images into context.
Why it matters: Text-only compression does not address the visual information that a multi-modal RAG system may retrieve. - MMRAG example selection: Describe the related problem MMRAG addresses and what its reported comparison shows about selecting few-shot examples.
Why it matters: This example shows that which items enter a limited prompt can affect performance and category coverage.
Key terms: multi-modal RAG, visual tokens, visual blocks, text and image tensors, token clustering, random projection, MMRAG, few-shot examples, top-similarity, diversity with skip-step selection
13.8.4 Hardware-aware budgeting
The section describes two emerging ways to make token budgets responsive to model confidence or hardware-specific latency rather than a fixed compression ratio.
- Confidence-based generation budget: Explain how Adaptive Budget Forcing uses token entropy as a confidence proxy to decide when to stop generation early, instead of relying on a fixed budget.
Why it matters: It illustrates a budget that can adapt during generation rather than being fixed in advance. - Hardware-specific latency target: Describe how HAP-E uses a latency predictor to meet a millisecond-per-token target on specific hardware, and why this differs from optimizing a compression ratio.
Why it matters: A smaller model or prompt is not necessarily the right optimization target when deployment latency depends on the device.
Key terms: Adaptive Budget Forcing (ABF), token entropy, confidence proxy, HAP-E (Hessian-Aware Pruning), latency predictor, millisecond-per-token target, memory bandwidth
13.8.5 Agentic self-compression
This subsection introduces agentic self-compression as on-demand context pruning during reasoning and explains its benefit and latency cost.
- Agentic self-compression tradeoff: Explain how an agent can decide to prune its context during reasoning instead of relying on one upfront compression step, and why that helps multi-hop tasks but adds latency.
Why it matters: It offers a way to adjust context as a task unfolds, provided the extra tool-call latency is acceptable.
Key terms: agentic self-compression, Python sandbox, context window, multi-hop tasks, just-in-time compression, upfront pruning
13.9 Summary
Where this is used later
- The reader is the bottleneck (chapter 15) · Introduces context compression as an approach to reducing material passed to the generator.
- Instrument everything (chapter 15) · Informs what to record when instrumenting a production RAG pipeline.
- Implementing an evaluation pipeline (chapter 14) · Evaluating pipeline changes depends on comparing answer-quality measurements.
- The efficiency-quality tradeoff (chapter 15) · Provides an example of an efficiency–quality tradeoff in RAG.
Concepts explained in this chapter
- LongLLMLinguaLongLLMLingua is a prompt-compression method that uses question-conditioned perplexity to remove less informative tokens while preserving information useful for downstream generation.
- compression ratioA compression ratio is the proportion or factor by which an input representation is reduced in size relative to the original.
- compactionContext compression reduces retrieved or prompt context while retaining information relevant to the query and generation task.
- AttentionRAGAttentionRAG is a retrieval-context compression technique that uses a transformer’s attention to select relevant tokens and sentences before generation.
- contrastive perplexityContrastive perplexity is a query-conditioned relevance measure based on the reduction in a token's perplexity after the user question is added to its conditioning context.
- pruningPruning is the removal of less relevant branches or items from a candidate set during retrieval.
- surrogate modelA proxy model is an approximate model used to estimate another model's behavior or to provide computationally cheaper scores for an input.
- retrieval rankingRanking is the ordering or filtering of retrieved candidates according to a relevance score, including ordering passages by relevance to a query.
- native pruningNative pruning is token or context pruning that derives importance scores directly from a model's internal attention weights.
- kv-cache evictionKey-value cache eviction is the removal of stored transformer attention states during decoding to control memory use.
- CORECORE is a retrieval-augmented generation compression framework that trains a compressor with reinforcement learning against downstream task rewards.
- RECOMPRECOMP is a retrieval-augmented generation framework that trains compressor models to optimize downstream task performance and selectively omit unhelpful retrieved context.
- abstractive compressorAn abstractive compressor generates a shorter passage that preserves important information from source text rather than merely selecting source sentences.
- attention dilutionAttention dilution is the spreading of attention probability mass across more tokens as a sequence becomes longer.
- query misalignmentQuery misalignment is the failure of a multi-token query to produce a single clear attention focus because its tokens attend to different context regions.
- LLMLinguaLLMLingua is a prompt-compression framework and software library for reducing prompt length while preserving task-relevant information.
- LLMLingua-2LLMLingua-2 is a prompt-compression system that uses a distilled token classifier to remove less important tokens.
- causal language modelA causal language model predicts each token from preceding tokens and is trained for autoregressive sequence modeling.
- coarse document rankingCoarse document ranking orders whole documents by estimated relevance before finer-grained compression.
- dynamic token budgetingDynamic token budgeting allocates different compression budgets to items according to their estimated importance or rank.
- abstractive summarizationAbstractive synthesis generates a shorter novel representation by integrating and rewording information from source documents.
- extractive summarizationExtractive pruning reduces context by deleting selected material without generating replacement text.
- proxy perplexityProxy perplexity is a prompt-compression approach that uses a language model's perplexity as a proxy for token informativeness.
- key-value cacheA decoder cache stores key and value attention states from previously processed tokens for reuse during autoregressive generation.
- extractive compressorAn extractive compressor is a compressor model that selects and combines useful sentences from retrieved documents without generating new wording.
- perplexityAbsolute perplexity is a token-predictability measure computed from general language statistics without conditioning on the user's question or task.
- hierarchical summarizationHierarchical summarization is summarization performed across multiple levels of organization, from granular items to broader aggregates.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns a policy from reward signals produced by interactions with an environment.
- information retrievalRetrieval finds candidate documents or passages from a knowledge base in response to a query.
- iterative retrievalIterative retrieval is a retrieval strategy that performs multiple retrieval steps, using intermediate results or reasoning to formulate subsequent searches.
- position biasPosition bias is an evaluation bias in which a judge favors whichever option appears first in a comparison.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
