A retrieved passage can contain the answer yet lose it during compression. Pruning by absolute perplexity favors unusual tokens and can discard ordinary-looking answer text while retaining surprising but irrelevant words. Contrastive perplexity instead asks whether the user’s question makes each token more predictable. section 13.2.1 section 13.2.2
Score a token by subtracting its perplexity when conditioned on the question from its perplexity given only preceding tokens. A large positive difference is a keep signal; a score near zero suggests the question adds little reason to retain it. section 13.2.2
Finding the dose, not the unusual word
For “what dose of aspirin was prescribed?”, the token “81” in “81 mg of aspirin daily” might drop from perplexity 5 to 1 when the question is supplied, giving a score near 4. A common “the” changes little and scores near zero. An off-topic medical term can have high absolute perplexity but still score low if the question does not make it more predictable. section 13.2.2
In LongLLMLingua, this token score is one stage of a larger process. Document-level filtering first removes irrelevant passages; reordering places strong passages at prompt boundaries; iterative pruning then applies contrastive perplexity within 200-token segments. A final recovery pass aligns damaged entity fragments in the generated answer with the uncompressed source. section 13.2.3
Listing 13.1 LLMLingua.
from llmlingua import PromptCompressor
from typing import List, Dict, Any
def compress_with_longllmlingua(
query: str,
documents: List[str],
rate: float = 0.5
) -> Dict[str, Any]:
"""Compress documents with contrastive perplexity."""
compressor = PromptCompressor( #A
model_name="Qwen/Qwen2.5-0.5B",
use_llmlingua2=False,
device_map="cpu"
)
compressed_output = compressor.compress_prompt( #B
documents,
question=query,
rate=rate,
condition_compare=True,
condition_in_question="after",
reorder_context="sort",
dynamic_context_compression_ratio=0.4,
rank_method="longllmlingua"
)
return compressed_outputThe Python example calls llmlingua’s PromptCompressor with a question, documents, and a compression rate. Its use_llmlingua2=False setting selects the question-conditioned LongLLMLingua pipeline; the alternative LLMLingua-2 classifier does not see the question. section 13.2.4
Use the extra proxy-model scoring when retrieved context is long, costly, or vulnerable to lost-in-the-middle errors—not merely because compression is available. The proxy pass adds latency for short, well-targeted documents. Inspect retained answer evidence as well as token savings: aggressive compression can remove meaning. section 13.2.4 section 13.9
