Perplexity
Perplexity is a language-model metric derived from the probabilities assigned to a sequence, indicating how well the model predicts it, with lower values generally reflecting better predictive fit.
Build a Large Language Model (From Scratch)
An evaluation metric defined as the exponential of cross entropy loss, indicating how uncertain the model is about the next token.
- How it is used
- Evaluates language-model performance and communicates next-token uncertainty alongside cross entropy loss.
- How deep it goes
- Explains interpretation and computes perplexity as torch.exp(loss), but does not detail broader dataset evaluation.
- Examples
- Uses two short token batches, tensor shapes, and a loss of 10.7940 yielding perplexity 48,725.8203.
- What you can do after
- Evaluate model uncertainty by converting a computed cross entropy loss into perplexity with PyTorch.
Read how Build a Large Language Model (From Scratch) explains perplexity →
Sutskever's List
Perplexity is a language-model metric representing the effective number of plausible next tokens, with lower values indicating greater predictive certainty.
- How it is used
- Uses predictive uncertainty to compare language-model performance on sequential modeling benchmarks.
- How deep it goes
- Defines the metric intuitively and compares benchmark values, but does not derive its mathematical formula or computation.
- Examples
- Uses a multiple-choice quiz, a 50,000-token random baseline, and a perplexity-100 Penn Treebank threshold.
- What you can do after
- Evaluate language-model certainty and compare sequential-modeling results using perplexity values.
Transformers in Action
Perplexity is a language-modeling metric equal to the exponential of average cross-entropy loss, measuring how well a model predicts a target token distribution.
- How it is used
- Evaluates language-model fit to a target dialogue distribution before and after supervised fine-tuning.
- How deep it goes
- Defines the metric and works through tokenization, padding, loss masking, batching, averaging, and exponentiation.
- Examples
- Compares base and fine-tuned Gemma checkpoints on up to 500 chat-formatted evaluation examples.
- What you can do after
- Diagnose/debug with it: Compare checkpoint fit using a batched PyTorch calculation that ignores padded tokens.
Generative AI in Action
Perplexity is a language-model metric that quantifies uncertainty in predicting the next token of a sequence.
- How it is used
- Evaluates language-model prediction confidence and adaptation after fine-tuning.
- How deep it goes
- Defines next-token uncertainty and score interpretation without deriving the metric mathematically.
- Examples
- Uses before-and-after fine-tuning comparisons across new text styles or domains.
- What you can do after
- Evaluate fine-tuned language-model improvements using confidence and prediction accuracy as signals.
Domain-Specific Small Language Models
A measure of how well a language model predicts tokens in sequence, with lower values generally indicating better predictive fit.
- How it is used
- Evaluates language-model predictive fit during inference testing and compares performance after quantization.
- How deep it goes
- Defines word, byte, and bits-per-byte measures with one concrete benchmark, but does not derive the metric mathematically.
- Examples
- Uses wikitext evaluation for GPT-Neo 2.7B and protein-sequence scoring for ProtGPT2 and antibody-generation models.
- What you can do after
- Evaluate language-model predictions and inspect whether quantization preserves comparable performance.
Read how Domain-Specific Small Language Models explains perplexity →
Related concepts
- input reversalproduced by · Input reversal is the technique of presenting a source sequence to a sequence model in reverse element order.
- language modelmeasures · A language model is a computational model that represents patterns in linguistic tokens to estimate their probabilities and predict or generate sequences of them.
- fluencymeasures · Fluency is the degree to which language production is smooth, coherent, grammatical, natural, and readable.
- compressioncontrasts with · Compression is the transformation of information into a more compact representation by encoding redundancy or omitting detail while retaining information needed for recovery or use.
- quantizationcontrasts with · Quantization is the mapping of numerical values, such as model parameters or activations, to a finite set of lower-precision representations.
- cross entropy lossuses · A loss function that measures the discrepancy between a target probability distribution and predicted probabilities, commonly through the negative log probability assigned to target outcomes.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.