Perplexity measures how well a language model predicts a target token distribution. It equals the exponential of average cross-entropy loss, so lower values indicate better fit and less surprise on the evaluation data. section 7.4
Perplexity
Perplexity is a language-modeling metric equal to the exponential of average cross-entropy loss, measuring how well a model predicts a target token distribution.
How Transformers in Action covers it
- How deep it goes
- Defines the metric and works through tokenization, padding, loss masking, batching, averaging, and exponentiation.
- Examples
- Compares base and fine-tuned Gemma checkpoints on up to 500 chat-formatted evaluation examples.
- What it assumes
- Assumes language-model evaluation, tokenization, chat templates, and model loss interfaces.
- What you can do after
- Diagnose/debug with it: Compare checkpoint fit using a batched PyTorch calculation that ignores padded tokens.
- How it connects
- Connects language modeling, cross-entropy loss, tokenization, chat templates, causal language models, and supervised fine-tuning.
Also explained in
-
Build a Large Language Model (From Scratch)An evaluation metric defined as the exponential of cross entropy loss, indicating how uncertain the model is about the next token.
-
Sutskever's ListPerplexity is a language-model metric representing the effective number of plausible next tokens, with lower values indicating greater predictive certainty.
-
Generative AI in ActionPerplexity is a language-model metric that quantifies uncertainty in predicting the next token of a sequence.
-
Domain-Specific Small Language ModelsA measure of how well a language model predicts tokens in sequence, with lower values generally indicating better predictive fit.
Compare how 5 books explain perplexity →
Related concepts
- input reversalproduced by · Input reversal is the technique of presenting a source sequence to a sequence model in reverse element order.
- language modelmeasures · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- fluencymeasures · Fluency is the degree to which language production is smooth, coherent, grammatical, natural, and readable.
- compressioncontrasts with · Compression is the transformation of information into a more compact representation to reduce storage or transmission requirements while retaining information needed for recovery or use.
- quantizationcontrasts with · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
- cross entropy lossuses · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.