LLMLingua compresses retrieved prompts before generation, reducing token cost and context pressure while preserving task-relevant information. A small proxy model scores and prunes text; a separate target generator answers from the compressed prompt. The proxy never generates, and the target never receives raw documents. section 13.2.4
Choose the pipeline according to the required signal:
- LongLLMLingua uses contrastive perplexity, attaches the question during scoring, and supports context reordering and dynamic compression. section 13.2.4
- LLMLingua-2 uses a BERT-style token classifier distilled from GPT-4 decisions; it is faster and tidier but does not see the question. section 13.2.4
- Avoid the extra proxy-model latency when retrieved documents are already short and well targeted. section 13.2.4
Compression can change the answer
A contract example shrinks 347 tokens to 212, or 1.6x. The compressed text drops wording and alters key clauses; one query therefore changes from correctly excluding indemnification obligations from a liability cap to incorrectly including them. section 13.2.4
Listing 13.1 LLMLingua.
from llmlingua import PromptCompressor
from typing import List, Dict, Any
def compress_with_longllmlingua(
query: str,
documents: List[str],
rate: float = 0.5
) -> Dict[str, Any]:
"""Compress documents with contrastive perplexity."""
compressor = PromptCompressor( #A
model_name="Qwen/Qwen2.5-0.5B",
use_llmlingua2=False,
device_map="cpu"
)
compressed_output = compressor.compress_prompt( #B
documents,
question=query,
rate=rate,
condition_compare=True,
condition_in_question="after",
reorder_context="sort",
dynamic_context_compression_ratio=0.4,
rank_method="longllmlingua"
)
return compressed_outputA Python listing shows PromptCompressor configuration, question-conditioned compression, context reordering, and result analysis. section 13.2.4
