Proxy perplexity compresses prompts by using a small language model to estimate which tokens carry information. It drops predictable filler, repeated connectives, and boilerplate while preserving surprising tokens such as named entities, numbers, and topical content. This makes compression practical even when the final generator is a closed API. section 13.1.1 section 13.2
Absolute perplexity scores tokens without considering the user's question, so it can remove predictable language that is nevertheless the requested answer—for example, a standard termination clause. LongLLMLingua improves the signal by conditioning perplexity on the question, making pruning query-aware while retaining a small, black-box-friendly proxy. section 13.1.1 section 13.2
Use proxy perplexity when the deployment needs prompt compression without hosting or fine-tuning the final generator:
- Choose it for closed-source APIs; the compressor can run as a separate, inexpensive open-weight scorer. section 13.2 section 13.7
- Expect moderate compute overhead and roughly 4x–10x compression, with low-to-moderate semantic-loss risk. section 13.7
- Protect contractual parties, drug names, model numbers, statute identifiers, and negations with force-token or must-keep lists. section 13.7
- Adjust the compression ratio to document density and API constraints rather than treating it as fixed. section 13.7
Why query conditioning matters
After “I sat down on the,” a language model finds “chair,” “couch,” or “floor” predictable; after “the patient was diagnosed with,” many conditions remain possible. Proxy perplexity tends to retain the latter's topical information, but question-conditioned scoring can also preserve predictable text when the question makes it relevant. section 13.2 section 13.1.1
Table 13.4 Method tradeoffs
|
Compression paradigm
|
Primary methodology
|
Optimal ratio
|
Compute overhead
|
Risk of semantic loss
|
|
Heuristic Pruning
|
TF-IDF Filtering
|
Low (~2x)
|
Negligible
|
Moderate
|
|
Proxy Perplexity
|
LongLLMLingua
|
Moderate (4x - 10x)
|
Moderate
|
Low to Moderate
|
|
Native Pruning
|
AttentionRAG
|
Moderate (4x - 6x)
|
Varies
|
Very Low
|
|
Trained Synthesis
|
RECOMP, CORE
|
Extreme (10x - 30x)
|
High
|
Low
|
|
KV-Cache Eviction
|
H2O, SnapKV, RazorAttention
|
N/A (decoding-time)
|
Low
|
Very Low
|
Compare proxy perplexity with heuristic pruning, native attention, trained synthesis, and KV-cache eviction across compression ratio, compute overhead, and semantic-loss risk. section 13.7
