liveBook · Manning

Perplexity

As explained in Rearchitecting LLMs

Perplexity is a language-modeling metric that measures how well a model predicts a sequence, with lower values indicating better predictive performance.

Perplexity (PPL) measures how well a language model predicts a sequence: lower values mean the model is less surprised by the observed words. It is useful for comparing pruning strategies and measuring whether a model retains general text-generation ability after compression. section 5.2.5 section 6.1.3

For depth pruning, measure the original model on a calibration dataset, then temporarily remove each block in turn and measure perplexity again. A large increase suggests the removed block is important; a small increase suggests it is more expendable. This requires one inference run per candidate block, making it costly for a model with many blocks. section 4.4

For width pruning, evaluate each variant on each target dataset as well as the unpruned baseline. Use the same dataset when comparing models: calibration can preserve capabilities that matter more for one domain than another. section 5.2.5

metrics_base_wiki = evaluate_metrics(model, dataloaderwiki)
metrics_base_sms = evaluate_metrics(model, dataloadersms)
 
metrics_wiki_wiki = evaluate_metrics(wiki_model, dataloaderwiki)
metrics_wiki_sms = evaluate_metrics(wiki_model, dataloadersms)
 
metrics_sms_wiki = evaluate_metrics(sms_model, dataloaderwiki) 
metrics_sms_sms = evaluate_metrics(sms_model, dataloadersms)

This evaluation matrix runs the base, Wiki-calibrated, and SMS-calibrated models against both datasets using evaluate_metrics. section 5.2.5

Table 5.1 Perplexity results
Model
Parameters
Wiki
Perplexity
SMS
Perplexity
Base
1.24B
25.69
122.91
Wiki model
1.07B
(~87%)
36.28
182.54
SMS model
1.07B
(~87%)
48.65
165.54
Static 20% pruned
1.07B
(~87%)
52
217

The Wiki-calibrated model scores better than the SMS-calibrated model on WikiText, while the SMS-calibrated model scores better on SMS. Both remain worse than the unpruned base on these measurements. section 5.2.5

Measure recovery after pruning

After four blocks were removed, one data-driven candidate had perplexity 126, versus 13.38 for the teacher. Basic knowledge distillation with 2,000 samples reduced the candidate’s perplexity to 18.9. Comparing both stages reveals initial degradation and recovery, not just the final score. section 6.1.3

Perplexity is not a substitute for task-specific evaluation. Pair it with capability benchmarks for a generalist model; for a tool-calling application, tool-selection accuracy may be a more appropriate main metric. A model optimized for another task can have worse perplexity while performing that task better. section 6.1.3 section 5.2.5

How Rearchitecting LLMs covers it

How deep it goes
Explains evaluation comparisons and block-removal costs without specifying a perplexity calculation formula.
Examples
Compares WikiText and SMS specialization and tracks a pruned candidate before and after distillation.
What you can do after
Evaluate pruning and recovery with perplexity alongside baselines and task-specific benchmarks.
How it connects
Connects perplexity to depth pruning, width pruning, knowledge distillation, and task-specific benchmarks.

Also explained in

Compare how 9 books explain perplexity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.