Perplexity measures how surprised a language model is by new tokens: lower values indicate greater predictive certainty, while higher values indicate uncertainty. section 4.3
A multiple-choice interpretation
A perplexity of 100 means the model behaves roughly as if it were choosing among 100 plausible next tokens. Randomly guessing from a 50,000-token vocabulary would yield perplexity around 50,000, so 100 reflects learned patterns despite limited confidence. section 4.3
Perplexity became a practical hurdle for recurrent neural networks on Penn Treebank benchmarks, where reducing it below 100 was once difficult because of overfitting. section 4.3
