A surrogate model is an approximate proxy that estimates another model’s behavior or supplies cheaper scores. In context compression, a small proxy scores retrieved text while a separate target generator produces the answer. section 13.2.4 section 13.0
The proxy never generates the response, and the target generator never receives the raw documents. This keeps compression upstream and permits a closed-source API to serve as the target. section 13.2.4
Use a surrogate model when the scoring cost is justified by the downstream savings:
- Compress retrieved context before generation to reduce token costs and address lost-in-the-middle effects. section 13.0
- Score tokens against the user’s query so predictable or irrelevant text can be discarded while answer-bearing content is retained. section 13.0
- Avoid the approach when documents are already short and targeted, because proxy inference can add unrecoverable latency. section 13.2.4
Legal-document compression
A contract example shrinks 347 tokens to 212, a 1.6x ratio, but the compressed text damages the indemnification evidence and produces an incorrect liability-cap answer. A tidier LLMLingua-2 output preserves the relevant limitation language and yields the correct answer, illustrating why compression rate must be evaluated with answer quality. section 13.2.4
Listing 13.1 LLMLingua.
from llmlingua import PromptCompressor
from typing import List, Dict, Any
def compress_with_longllmlingua(
query: str,
documents: List[str],
rate: float = 0.5
) -> Dict[str, Any]:
"""Compress documents with contrastive perplexity."""
compressor = PromptCompressor( #A
model_name="Qwen/Qwen2.5-0.5B",
use_llmlingua2=False,
device_map="cpu"
)
compressed_output = compressor.compress_prompt( #B
documents,
question=query,
rate=rate,
condition_compare=True,
condition_in_question="after",
reorder_context="sort",
dynamic_context_compression_ratio=0.4,
rank_method="longllmlingua"
)
return compressed_outputInspect a complete PromptCompressor setup using a small Qwen proxy, query-conditioned scoring, context reordering, and a configurable compression rate. section 13.2.4






