liveBook · Manning

Surrogate model

As explained in Retrieval Augmented Generation, The Foundational Ideas

A proxy model is an approximate model used to estimate another model's behavior or to provide computationally cheaper scores for an input.

A surrogate model is an approximate proxy that estimates another model’s behavior or supplies cheaper scores. In context compression, a small proxy scores retrieved text while a separate target generator produces the answer. section 13.2.4 section 13.0

The proxy never generates the response, and the target generator never receives the raw documents. This keeps compression upstream and permits a closed-source API to serve as the target. section 13.2.4

Use a surrogate model when the scoring cost is justified by the downstream savings:

  • Compress retrieved context before generation to reduce token costs and address lost-in-the-middle effects. section 13.0
  • Score tokens against the user’s query so predictable or irrelevant text can be discarded while answer-bearing content is retained. section 13.0
  • Avoid the approach when documents are already short and targeted, because proxy inference can add unrecoverable latency. section 13.2.4

Legal-document compression

A contract example shrinks 347 tokens to 212, a 1.6x ratio, but the compressed text damages the indemnification evidence and produces an incorrect liability-cap answer. A tidier LLMLingua-2 output preserves the relevant limitation language and yields the correct answer, illustrating why compression rate must be evaluated with answer quality. section 13.2.4

Listing 13.1 LLMLingua.
from llmlingua import PromptCompressor
from typing import List, Dict, Any
 
def compress_with_longllmlingua(
    query: str,
    documents: List[str],
    rate: float = 0.5
) -> Dict[str, Any]:
    """Compress documents with contrastive perplexity."""
 
    compressor = PromptCompressor(  #A
        model_name="Qwen/Qwen2.5-0.5B",
        use_llmlingua2=False,
        device_map="cpu"
    )
 
    compressed_output = compressor.compress_prompt(  #B
        documents,
        question=query,
        rate=rate,
        condition_compare=True,
        condition_in_question="after",
        reorder_context="sort",
        dynamic_context_compression_ratio=0.4,
        rank_method="longllmlingua"
    )
 
    return compressed_output

Inspect a complete PromptCompressor setup using a small Qwen proxy, query-conditioned scoring, context reordering, and a configurable compression rate. section 13.2.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the two-model split and operational tradeoffs, but stops short of deriving the scoring algorithms.
Examples
Demonstrates LLMLingua code, token-count analysis, benchmark results, and a legal indemnification case.
What it assumes
Assumes retrieval and generation concepts while establishing context compression and proxy scoring within that pipeline.
What you can do after
Apply it via LLMLingua to compress retrieved documents and inspect token reduction, latency, and answer-preservation tradeoffs.
How it connects
Connects surrogate models to context compression, retrieval-augmented generation, token pruning, and KV-cache eviction.

Also explained in

Compare how 7 books explain surrogate model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.