liveBook · Manning

Proxy perplexity

As explained in Retrieval Augmented Generation, The Foundational Ideas

Proxy perplexity is a prompt-compression approach that uses a language model's perplexity as a proxy for token informativeness.

Proxy perplexity compresses prompts by using a small language model to estimate which tokens carry information. It drops predictable filler, repeated connectives, and boilerplate while preserving surprising tokens such as named entities, numbers, and topical content. This makes compression practical even when the final generator is a closed API. section 13.1.1 section 13.2

Absolute perplexity scores tokens without considering the user's question, so it can remove predictable language that is nevertheless the requested answer—for example, a standard termination clause. LongLLMLingua improves the signal by conditioning perplexity on the question, making pruning query-aware while retaining a small, black-box-friendly proxy. section 13.1.1 section 13.2

Use proxy perplexity when the deployment needs prompt compression without hosting or fine-tuning the final generator:

  • Choose it for closed-source APIs; the compressor can run as a separate, inexpensive open-weight scorer. section 13.2 section 13.7
  • Expect moderate compute overhead and roughly 4x–10x compression, with low-to-moderate semantic-loss risk. section 13.7
  • Protect contractual parties, drug names, model numbers, statute identifiers, and negations with force-token or must-keep lists. section 13.7
  • Adjust the compression ratio to document density and API constraints rather than treating it as fixed. section 13.7

Why query conditioning matters

After “I sat down on the,” a language model finds “chair,” “couch,” or “floor” predictable; after “the patient was diagnosed with,” many conditions remain possible. Proxy perplexity tends to retain the latter's topical information, but question-conditioned scoring can also preserve predictable text when the question makes it relevant. section 13.2 section 13.1.1

Table 13.4 Method tradeoffs
Compression paradigm
Primary methodology
Optimal ratio
Compute overhead
Risk of semantic loss
Heuristic Pruning
TF-IDF Filtering
Low (~2x)
Negligible
Moderate
Proxy Perplexity
LongLLMLingua
Moderate (4x - 10x)
Moderate
Low to Moderate
Native Pruning
AttentionRAG
Moderate (4x - 6x)
Varies
Very Low
Trained Synthesis
RECOMP, CORE
Extreme (10x - 30x)
High
Low
KV-Cache Eviction
H2O, SnapKV, RazorAttention
N/A (decoding-time)
Low
Very Low

Compare proxy perplexity with heuristic pruning, native attention, trained synthesis, and KV-cache eviction across compression ratio, compute overhead, and semantic-loss risk. section 13.7

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains token scoring, query conditioning, tradeoffs, safeguards, and tuning, but does not detail implementation internals.
Examples
Uses predictable connective words, clinical diagnoses, contract clauses, named entities, numbers, and force-token safeguards.
What it assumes
Assumes familiarity with language-model tokens and prompts while defining perplexity and contrasting query-agnostic and query-aware scoring.
What you can do after
Apply it via LongLLMLingua or similar methods, selecting ratios and must-keep tokens for production prompt compression.
How it connects
Connects perplexity with prompt compression, LongLLMLingua, native pruning, attention, and KV-cache eviction.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.