liveBook · Manning

Perplexity

As explained in Build a Large Language Model (From Scratch)

An evaluation metric defined as the exponential of cross entropy loss, indicating how uncertain the model is about the next token.

Perplexity evaluates next-token prediction uncertainty by exponentiating cross entropy loss. Lower perplexity means the predicted distribution more closely matches the dataset distribution. section 5.1.2

Compute perplexity from model outputs in three stages:

  • Obtain logits shaped [batch, sequence, vocabulary] and align them with target token IDs. section 5.1.2
  • Calculate cross entropy over flattened logits and targets, combining token positions across the batch. section 5.1.2
  • Apply torch.exp(loss) to convert the loss into a more interpretable uncertainty measure. section 5.1.2

Worked value

A cross entropy loss of approximately 10.7940 produces perplexity 48,725.8203. This indicates uncertainty equivalent to choosing among roughly 48,725 vocabulary tokens at each step. section 5.1.2

How Build a Large Language Model (From Scratch) covers it

How deep it goes
Explains interpretation and computes perplexity as torch.exp(loss), but does not detail broader dataset evaluation.
Examples
Uses two short token batches, tensor shapes, and a loss of 10.7940 yielding perplexity 48,725.8203.
What it assumes
Assumes cross entropy, logits, target token IDs, and probability distributions while demonstrating their alignment from tensor outputs.
What you can do after
Evaluate model uncertainty by converting a computed cross entropy loss into perplexity with PyTorch.
How it connects
Connects cross entropy loss, language modeling, next-token prediction, logits, softmax, and vocabulary distributions.
Cover of Build a Large Language Model (From Scratch)
liveBookmanning.com
Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

Compare how 5 books explain perplexity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.