liveBook · Manning

Compaction

As explained in Retrieval Augmented Generation, The Foundational Ideas

Context compression reduces retrieved or prompt context while retaining information relevant to the query and generation task.

A retriever can find the right passage and still produce a poor answer. When that passage is buried among long, noisy results, the generator may miss it while processing more tokens at greater cost and latency. Context compression reduces what the generator must read, usually after retrieval and before generation. section 13.0 section 13.1

The simplest approach removes irrelevant passages. More selective approaches retain relevant documents but shorten them: extractive pruning deletes tokens, while abstractive compression rewrites passages. LongLLMLingua uses a small proxy model to judge which documents and tokens matter to the question; it can run alongside a closed-source generator without fine-tuning that generator. section 13.0 section 13.2

LongLLMLingua compresses a retrieved prompt in four stages:

  • Score whole documents against the question and discard weak matches before token-level work. section 13.2.3
  • Put the highest-ranked document first and the second-highest last, where answer extraction tends to be stronger than in the middle. section 13.2.3
  • Prune lower-value tokens in the survivors using query-conditioned scores, then align fragmented entities in the output with the original source. section 13.2.3

Legal-document assistant

A supply-agreement clause retrieved at position six was overlooked in a context exceeding 7,000 tokens. At 4× compression, the relevant chunk received more budget and moved near the front, improving extraction. At 6×, pruning removed a liability-cap sub-clause and its article number; the generator invented the missing detail. The team reduced compression until citations returned and exempted the top-ranked document from token-level pruning. section 13.6

Table 13.4 Method tradeoffs
Compression paradigm
Primary methodology
Optimal ratio
Compute overhead
Risk of semantic loss
Heuristic Pruning
TF-IDF Filtering
Low (~2x)
Negligible
Moderate
Proxy Perplexity
LongLLMLingua
Moderate (4x - 10x)
Moderate
Low to Moderate
Native Pruning
AttentionRAG
Moderate (4x - 6x)
Varies
Very Low
Trained Synthesis
RECOMP, CORE
Extreme (10x - 30x)
High
Low
KV-Cache Eviction
H2O, SnapKV, RazorAttention
N/A (decoding-time)
Low
Very Low

Compare compression methods by typical ratio, compute overhead, and risk of semantic loss before choosing a pipeline. section 13.7

Treat the compression ratio as a setting to evaluate, not a target to maximize. Start around 4×, compare answers with an uncompressed baseline on held-out queries, and back off if quality falls. Check domain-critical terms, dates, negation, and citations: deleting a few tokens can reverse a clause. Prefer a simple retrieve-and-generate pipeline when long or noisy context is not a measured problem. KV-cache eviction addresses a different cost—the memory used during decoding rather than the input prompt. section 13.7 section 13.8.2 section 15.5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Works through document filtering, reordering, token pruning, and output recovery, then gives a baseline comparison and ratio-selection checklist.
Examples
Uses a legal assistant whose relevant clause was overlooked, then contrasts a successful 4× setting with a damaging 6× setting.
What it assumes
Assumes a working retrieval-and-generation pipeline but explains the perplexity signal used for query-conditioned pruning.
What you can do after
Operate/configure a compression pipeline by testing answer quality against an uncompressed baseline and backing off when critical details disappear.
How it connects
Connects retrieval-augmented generation and reranking to perplexity, attention bias, abstractive compression, and KV-cache eviction.

Also explained in

Compare how 10 books explain compaction →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.