A retriever can find the right passage and still produce a poor answer. When that passage is buried among long, noisy results, the generator may miss it while processing more tokens at greater cost and latency. Context compression reduces what the generator must read, usually after retrieval and before generation. section 13.0 section 13.1
The simplest approach removes irrelevant passages. More selective approaches retain relevant documents but shorten them: extractive pruning deletes tokens, while abstractive compression rewrites passages. LongLLMLingua uses a small proxy model to judge which documents and tokens matter to the question; it can run alongside a closed-source generator without fine-tuning that generator. section 13.0 section 13.2
LongLLMLingua compresses a retrieved prompt in four stages:
- Score whole documents against the question and discard weak matches before token-level work. section 13.2.3
- Put the highest-ranked document first and the second-highest last, where answer extraction tends to be stronger than in the middle. section 13.2.3
- Prune lower-value tokens in the survivors using query-conditioned scores, then align fragmented entities in the output with the original source. section 13.2.3
Legal-document assistant
A supply-agreement clause retrieved at position six was overlooked in a context exceeding 7,000 tokens. At 4× compression, the relevant chunk received more budget and moved near the front, improving extraction. At 6×, pruning removed a liability-cap sub-clause and its article number; the generator invented the missing detail. The team reduced compression until citations returned and exempted the top-ranked document from token-level pruning. section 13.6
Table 13.4 Method tradeoffs
|
Compression paradigm
|
Primary methodology
|
Optimal ratio
|
Compute overhead
|
Risk of semantic loss
|
|
Heuristic Pruning
|
TF-IDF Filtering
|
Low (~2x)
|
Negligible
|
Moderate
|
|
Proxy Perplexity
|
LongLLMLingua
|
Moderate (4x - 10x)
|
Moderate
|
Low to Moderate
|
|
Native Pruning
|
AttentionRAG
|
Moderate (4x - 6x)
|
Varies
|
Very Low
|
|
Trained Synthesis
|
RECOMP, CORE
|
Extreme (10x - 30x)
|
High
|
Low
|
|
KV-Cache Eviction
|
H2O, SnapKV, RazorAttention
|
N/A (decoding-time)
|
Low
|
Very Low
|
Compare compression methods by typical ratio, compute overhead, and risk of semantic loss before choosing a pipeline. section 13.7
Treat the compression ratio as a setting to evaluate, not a target to maximize. Start around 4×, compare answers with an uncompressed baseline on held-out queries, and back off if quality falls. Check domain-critical terms, dates, negation, and citations: deleting a few tokens can reverse a clause. Prefer a simple retrieve-and-generate pipeline when long or noisy context is not a measured problem. KV-cache eviction addresses a different cost—the memory used during decoding rather than the input prompt. section 13.7 section 13.8.2 section 15.5.1









