Perplexity (PPL) measures how well a language model predicts a sequence: lower values mean the model is less surprised by the observed words. It is useful for comparing pruning strategies and measuring whether a model retains general text-generation ability after compression. section 5.2.5 section 6.1.3
For depth pruning, measure the original model on a calibration dataset, then temporarily remove each block in turn and measure perplexity again. A large increase suggests the removed block is important; a small increase suggests it is more expendable. This requires one inference run per candidate block, making it costly for a model with many blocks. section 4.4
For width pruning, evaluate each variant on each target dataset as well as the unpruned baseline. Use the same dataset when comparing models: calibration can preserve capabilities that matter more for one domain than another. section 5.2.5
metrics_base_wiki = evaluate_metrics(model, dataloaderwiki) metrics_base_sms = evaluate_metrics(model, dataloadersms) metrics_wiki_wiki = evaluate_metrics(wiki_model, dataloaderwiki) metrics_wiki_sms = evaluate_metrics(wiki_model, dataloadersms) metrics_sms_wiki = evaluate_metrics(sms_model, dataloaderwiki) metrics_sms_sms = evaluate_metrics(sms_model, dataloadersms)
This evaluation matrix runs the base, Wiki-calibrated, and SMS-calibrated models against both datasets using evaluate_metrics. section 5.2.5
Table 5.1 Perplexity results
|
Model
|
Parameters
|
Wiki
Perplexity
|
SMS
Perplexity
|
|
Base
|
1.24B
|
25.69
|
122.91
|
|
Wiki model
|
1.07B
(~87%)
|
36.28
|
182.54
|
|
SMS model
|
1.07B
(~87%)
|
48.65
|
165.54
|
|
Static 20% pruned
|
1.07B
(~87%)
|
52
|
217
|
The Wiki-calibrated model scores better than the SMS-calibrated model on WikiText, while the SMS-calibrated model scores better on SMS. Both remain worse than the unpruned base on these measurements. section 5.2.5
Measure recovery after pruning
After four blocks were removed, one data-driven candidate had perplexity 126, versus 13.38 for the teacher. Basic knowledge distillation with 2,000 samples reduced the candidate’s perplexity to 18.9. Comparing both stages reveals initial degradation and recovery, not just the final score. section 6.1.3
Perplexity is not a substitute for task-specific evaluation. Pair it with capability benchmarks for a generalist model; for a tool-calling application, tool-selection accuracy may be a more appropriate main metric. A model optimized for another task can have worse perplexity while performing that task better. section 6.1.3 section 5.2.5








