liveBook · Manning

Chapter 13: Context compression and pruning

What you will learn in chapter 13 of Retrieval Augmented Generation, The Foundational Ideas

This introduction explains why long retrieved context can hurt RAG answers and outlines ways to reduce or prioritize what the generator processes.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why long retrieved context can hurt RAG answers and outlines ways to reduce or prioritize what the generator processes.

Key terms: extraction failure, lost in the middle, context compression, signal-to-noise ratio, token-level pruning, trained abstractive compressors, KV-cache eviction, query, token budget, positional bias

13.1 The costs of long context

Explains how long retrieved contexts can weaken answer extraction and increase cost and latency, and why removing irrelevant tokens before generation helps.

Key terms: contextual dilution, self-attention, primacy bias, recency bias, parametric priors, NoLiMa, literal overlap, needle-in-a-haystack tests, input tokens, time-to-first-token

13.1.1 Evolution of context compression

This section traces context compression as a series of fixes for methods that discard useful answer evidence or misjudge what the generator needs.

Key terms: lexical heuristics, TF-IDF, BM25, proxy-model perplexity, absolute perplexity, query-agnostic, contrastive perplexity, LongLLMLingua, query-aware, proxy

13.1.2 Types of compression

This section classifies context-compression methods by what they remove and whether they need access to the generator’s internals.

Key terms: scope, token-level methods, sentence-level methods, document-level methods, access mode, black-box methods, white-box methods, closed API, open-weight LLM, attention weights

13.2 Perplexity-based compression with LongLLMLingua

The section explains why proxy-perplexity compression is practical and introduces the perplexity signal that LongLLMLingua conditions on the user's question.

Key terms: proxy-perplexity, LongLLMLingua, open-weight scorer, perplexity, token, low perplexity, high perplexity, user's question

13.2.1 Why absolute perplexity fails

This subsection explains why question-blind perplexity can remove useful retrieved text and contrasts it with question-aware selection.

Key terms: absolute perplexity, contrastive perplexity, high-entropy noise

13.2.2 Contrastive perplexity as a relevance signal

This section explains how LongLLMLingua uses a question-dependent change in token perplexity to decide which document tokens to keep.

Key terms: conditional perplexity, contrastive perplexity, user question, relevance signal, proxy model, 81 mg, contrastive score, thrombocytopenia, ground-truth answer

13.2.3 The LongLLMLingua pipeline

This section explains how LongLLMLingua filters, reorders, and prunes retrieved context, then repairs fragments in the generated answer.

Key terms: budget controller, coarse-grained compression, proxy language model, anchor sentence, perplexity, reordering, relevance scores, U-shaped attention curve, contrastive perplexity, query-aware

13.2.4 Implementing a prompt compressor with the LLMLingua

This section shows how to configure and assess LongLLMLingua prompt compression, using a legal-document example to expose both its benefits and its risks.

Key terms: proxy model, target generator, PromptCompressor, effective context window, use_llmlingua2=False, causal language model, condition_compare, reorder_context, dynamic_context_compression_ratio, coarse ranker

13.3 Native pruning with AttentionRAG

This section explains how AttentionRAG uses a small model’s attention weights to prune retrieved context, why it uses an answer-shaped prefix, and what its accuracy and deployment tradeoffs are.

Key terms: native pruning, attention weights, perplexity, alignment gap, attention dilution, query misalignment, answer hint prefix, chunks, attention features, all-layer aggregation

13.3.1 Beyond all-layer aggregation: Evaluator heads

This subsection explains how EHPC uses selected early attention heads to prune tokens before expensive layers, and when its speed advantage is worth the tradeoff against AttentionRAG.

Key terms: EHPC, evaluator heads, token importance, forward pass, AttentionRAG, all-layer sum, latency, throughput

13.4 Extensions: The RECOMP framework

This section explains how RECOMP goes beyond token pruning by training compressors for answer quality, allowing rewritten evidence, and withholding context when retrieval is unhelpful.

Key terms: RECOMP, token pruning, exact match, F1, extractive compressor, abstractive compressor, abstractive synthesis, selective augmentation, empty string, I don't know

13.4.1 CORE and multi-hop synthesis

This section explains how CORE trains task-oriented compression without human labels, why multi-hop tasks need synthesis rather than token pruning, and how to choose between extractive and abstractive compression.

Key terms: CORE, knowledge distillation warm-up, RL policy refinement, exact match, F1, static black-box environment, Group Relative Policy Optimization, GRPO, critic-free, group-normalized empirical advantage

13.5 Implementing a query-aware compressor

The section shows how to insert query-aware compression into a RAG pipeline and explains its effects on the query interface, token budget, and operational logging.

Key terms: query-aware compressor, retrieve-compress-generate topology, orchestrator, pruned payload, LlamaIndex, llmlingua, postprocessor, target_token, query engine, uncompressed prompts

13.5.1 Recursive retrieval and index nodes

This section explains how recursive retrieval uses compressed summaries and links to original document chunks to reduce context without losing hierarchical detail.

Key terms: chunk-level compression, hierarchical documents, relational meaning, recursive retrieval, LlamaIndex, IndexNode, index_id, compressed summaries, uncompressed child nodes

13.6 Case study: Legal document synthesis

Legal case studies show how compression can improve retrieval-augmented answers, where excessive pruning can damage them, and when latency or scale justifies the work.

Key terms: LongLLMLingua, failure point 5, compression ratio, failure point 4, contrastive perplexity, keep_first_sentence, token-level pruning, financial break-even point, time-to-first-token, hierarchical summarization

13.7 Tradeoffs: Compression ratio versus semantic loss

This section explains how to choose and validate a context-compression strategy that saves tokens without losing facts needed for accurate answers.

Key terms: Pareto frontier, compression ratio, semantic loss, noise removal zone, safe operating range, held-out query set, exact match, force tokens, must-keep strings, domain-critical terminology

13.8 Future directions and adjacent research

Introduces five emerging directions in context compression beyond runtime input-prompt compression.

13.8.1 Gist tokens and trained prompt compression

This subsection explains how trained gist tokens replace a long prompt and when that approach is practical.

Key terms: gist prefix, virtual continuous tokens, key/value activations, gist caching, instruction caching, compress-by-training, runtime pruning

13.8.2 KV-cache eviction

This section explains how KV-cache eviction reduces decoding memory, surveys four eviction approaches, and distinguishes when it is more useful than prompt compression.

Key terms: prompt compression, prefill cost, KV-cache eviction, decoding memory cost, H2O, SnapKV, RazorAttention, retrieval heads, compensation token, ManifoldKV

13.8.3 Multi-modal retrieval

This section explains why retrieval across text and images needs visual-token compression and describes approaches to representing visual documents and selecting informative examples.

Key terms: multi-modal RAG, visual tokens, visual blocks, text and image tensors, token clustering, random projection, MMRAG, few-shot examples, top-similarity, diversity with skip-step selection

13.8.4 Hardware-aware budgeting

The section describes two emerging ways to make token budgets responsive to model confidence or hardware-specific latency rather than a fixed compression ratio.

Key terms: Adaptive Budget Forcing (ABF), token entropy, confidence proxy, HAP-E (Hessian-Aware Pruning), latency predictor, millisecond-per-token target, memory bandwidth

13.8.5 Agentic self-compression

This subsection introduces agentic self-compression as on-demand context pruning during reasoning and explains its benefit and latency cost.

Key terms: agentic self-compression, Python sandbox, context window, multi-hop tasks, just-in-time compression, upfront pruning

13.9 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 12: Graph-based RAG Chapter 14: Evaluating RAG systems →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.