liveBook · Manning

Perplexity

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A language-model metric derived from the probability assigned to a sequence of text, with lower values indicating less model surprise.

Perplexity measures how surprised a language model is by a sequence of text. Lower values generally indicate more natural language because the model assigns higher probability to the sequence. section 1.1.2

Use it as a fast, inexpensive intrinsic metric for rapid iteration and regression testing. It evaluates how text is written, not whether the output completes its task, remains faithful to data, or satisfies application-specific requirements. section 1.1.2

Interpret perplexity alongside other evaluation signals:

  • Compare it with coherence and consistency checks when evaluating generated text. section 1.1.2
  • Add human evaluation or LLM-as-a-Judge when correctness, relevance, or task success matters. section 1.1.2
Figure 1.5 Reference-free evaluation workflow. Quality signals are defined for the task. The AI generates outputs for test inputs. Metrics assess each output’s intrinsic properties without reference answers. Scores reveal quality patterns and failure modes.

Places perplexity within reference-free evaluation, where intrinsic metrics assess generated outputs without reference answers and expose quality patterns or failure modes. section 1.1.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines surprise and practical interpretation, but stops short of presenting a calculation or implementation procedure.
Examples
Uses a language model’s prediction of a text sequence as the concrete evaluation object.
What you can do after
Evaluate/understand model-generated language quickly, while recognizing when task-specific or human judgment is still required.
How it connects
Links perplexity to intrinsic metrics, reference-free evaluation, coherence, consistency checks, and human evaluation.

Also explained in

Compare how 9 books explain perplexity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.