liveBook · Manning

Capability retention

As explained in Rearchitecting LLMs

Capability retention is the proportion of a model's evaluated performance preserved after a structural modification such as pruning.

Capability retention measures the proportion of evaluated model performance preserved after a structural modification, such as pruning blocks or distilling a smaller student model. It turns model compression into a controlled comparison: reduce size or computation, then check which capabilities remain. section 4.5 section 6.7

Use capability retention to make compression decisions reproducible:

  • Choose a baseline model and fixed evaluation benchmarks such as WinoGrande, Lambada, ARC-Easy, HellaSwag, or PIQA. section 4.5 section 6.7
  • Modify the architecture or training configuration, then evaluate the resulting model against the same baseline and tasks. section 4.5 section 6.7
  • Track retention as pruning becomes more aggressive, and inspect whether degradation remains gradual or reaches a capability-breaking point. section 4.5
  • Compare retention with operational measures such as perplexity, training time, and the resulting model's deployment suitability. section 6.7

Distilled student baseline

A 14-block distilled student reported 89.5% capability retention and 10.85 perplexity after training on 40K Cosmopedia samples for five epochs. The result becomes a target for changing loss weights, block selection, mapping, data composition, or training hyperparameters. section 6.7

Pruning retention curve

Remove 2, 3, 4, 5, and 6 blocks using WinoGrande importance scores, evaluate WinoGrande and Lambada retention, and plot both degradation curves to locate a breaking point. section 4.5

How Rearchitecting LLMs covers it

How deep it goes
Operationalizes retention through controlled benchmarks, progressive pruning curves, and configuration experiments, but does not define a universal aggregation formula.
Examples
Uses WinoGrande, Lambada, ARC-Easy, HellaSwag, PIQA, progressive block removal, and a 14-block distilled student.
What it assumes
Assumes a baseline model, configurable blocks or distillation pipeline, and access to benchmark evaluation and training workflows.
What you can do after
Evaluate/understand which compressed-model configurations preserve capabilities and where further modification causes unacceptable degradation.
How it connects
Connects model pruning, block selection, knowledge distillation, benchmark evaluation, and perplexity.

Also explained in

Compare how 2 books explain capability retention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.