A retrieved passage can contain the answer and still produce a poor answer: long prompts cost more, take longer to process, and can bury relevant text in the middle. LongLLMLingua compresses retrieved context before generation. A small proxy model decides what to retain; a separate generator receives the compressed prompt. Because compression happens upstream, the generator can be a closed-source API. section 13.0 section 13.2.4
Ordinary perplexity-based pruning drops tokens that are easy to predict from surrounding text. That can erase predictable wording that answers the user’s question. LongLLMLingua instead compares each token’s perplexity with and without the question. If the question makes a token substantially less surprising, its contrastive score signals relevance; tokens the question does not help predict are candidates for removal. section 13.1.1 section 13.2.2
A dose hidden among other details
For “What dose of aspirin was prescribed?”, the token “81” in “81 mg of aspirin daily” becomes more predictable when the question is supplied, so it receives a strong keep signal. A connective such as “the” changes little; an unrelated medical term can be surprising yet still score low because the dose question does not make it more predictable. section 13.2.2
The score subtracts question-conditioned token perplexity from perplexity without the question. A large positive difference favors keeping the token. section 13.2.2
Compression proceeds in four stages:
- Filter whole documents using a question-anchored proxy score, removing irrelevant documents before token-level work. section 13.2.3
- Reorder survivors so the highest-ranked passage appears first and the second-highest last, countering the tendency to overlook the middle of a prompt. section 13.2.3
- Prune tokens iteratively in 200-token segments using contrastive perplexity. section 13.2.3
- Align the generator’s output with the uncompressed source to repair fragmented entities. section 13.2.3
Listing 13.1 LLMLingua.
from llmlingua import PromptCompressor
from typing import List, Dict, Any
def compress_with_longllmlingua(
query: str,
documents: List[str],
rate: float = 0.5
) -> Dict[str, Any]:
"""Compress documents with contrastive perplexity."""
compressor = PromptCompressor( #A
model_name="Qwen/Qwen2.5-0.5B",
use_llmlingua2=False,
device_map="cpu"
)
compressed_output = compressor.compress_prompt( #B
documents,
question=query,
rate=rate,
condition_compare=True,
condition_in_question="after",
reorder_context="sort",
dynamic_context_compression_ratio=0.4,
rank_method="longllmlingua"
)
return compressed_outputThe Python example passes a question and documents to `PromptCompressor`. `use_llmlingua2=False` selects the question-conditioned LongLLMLingua path; `rate` sets the requested compression level. section 13.2.4
Algorithm selection matters. The same class also exposes LLMLingua-2, a faster token classifier that does not see the question. On a contract question about whether indemnification falls under a liability cap, question-aware compression preserves the exception and supports the correct “no”; the illustrated LLMLingua-2 output loses it and leads to “yes.” A shorter or neater prompt is not proof that the answer survived. section 13.2.4
Treat the compression ratio as an evaluated setting, not a target to maximize. Moderate compression can remove distracting text, but aggressive pruning can drop article numbers, dates, entities, or a modifier such as “Except” and reverse the meaning. Start around 4×, compare answers against an uncompressed baseline on held-out questions, and back off if accuracy falls. Compression is less attractive for already short, targeted retrieval: the proxy adds latency. In an interactive legal workflow, faster responses may justify it even when token savings alone do not cover pipeline maintenance. section 13.7 section 13.6 section 13.2.4
