liveBook · Manning

Perplexity

As explained in Retrieval Augmented Generation, The Foundational Ideas

Absolute perplexity is a token-predictability measure computed from general language statistics without conditioning on the user's question or task.

Perplexity measures how predictable each token is to a language model from the tokens before it. Low perplexity marks familiar, predictable language; high perplexity marks surprising content such as numbers, named entities, or specialized terms. section 13.2

Absolute perplexity ignores the user's question. In context compression, this can remove relevant sentences because generic wording looks predictable, while retaining complex but off-topic language because it appears surprising. section 13.2.1

Use perplexity in two contrasting ways:

  • Absolute perplexity scores tokens from general language statistics alone and can discard information needed for the task. section 13.2.1
  • Contrastive perplexity adds the user question and keeps tokens whose predictability improves substantially under that question. section 13.2.2
  • Perplexity Distillation uses the language model's signal to train a retriever to prioritize documents useful for generating the correct output. section 6.5

Question-conditioned token selection

For the question “what dose of aspirin was prescribed?”, the token “81” may have general perplexity around 5 but question-conditioned perplexity around 1, producing a strong keep signal. The connective “the” changes little and is pruned; unrelated “thrombocytopenia” remains surprising without becoming relevant and is also pruned. section 13.2.2

PPL(xᵢ | x<ᵢ)

Browse the token-level probability form, then compare it with the query-conditioned contrastive score. section 13.2.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines token-level prediction and contrasts query-free scoring with query-conditioned relevance, without detailing model implementation.
Examples
Uses predictable connectives, aspirin dosage, named entities, numbers, and unrelated medical terminology as scoring examples.
What it assumes
Assumes familiarity with language-model token probabilities and retrieval pipelines, while defining predictability from zero.
What you can do after
Diagnose why absolute-perplexity filtering removes relevant context and evaluate query-conditioned alternatives such as LongLLMLingua.
How it connects
Connects language models, retrieval-augmented generation, prompt compression, retrievers, and contrastive perplexity.

Also explained in

Compare how 9 books explain perplexity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.