Capability retention measures the proportion of evaluated model performance preserved after a structural modification, such as pruning blocks or distilling a smaller student model. It turns model compression into a controlled comparison: reduce size or computation, then check which capabilities remain. section 4.5 section 6.7
Use capability retention to make compression decisions reproducible:
- Choose a baseline model and fixed evaluation benchmarks such as WinoGrande, Lambada, ARC-Easy, HellaSwag, or PIQA. section 4.5 section 6.7
- Modify the architecture or training configuration, then evaluate the resulting model against the same baseline and tasks. section 4.5 section 6.7
- Track retention as pruning becomes more aggressive, and inspect whether degradation remains gradual or reaches a capability-breaking point. section 4.5
- Compare retention with operational measures such as perplexity, training time, and the resulting model's deployment suitability. section 6.7
Distilled student baseline
A 14-block distilled student reported 89.5% capability retention and 10.85 perplexity after training on 40K Cosmopedia samples for five epochs. The result becomes a target for changing loss weights, block selection, mapping, data composition, or training hyperparameters. section 6.7
Pruning retention curve
Remove 2, 3, 4, 5, and 6 blocks using WinoGrande importance scores, evaluate WinoGrande and Lambada retention, and plot both degradation curves to locate a breaking point. section 4.5

