liveBook · Manning

Capability retention

Capability retention is the proportion of evaluated capabilities or performance preserved by a modified or derived model relative to a reference model.

How 2 Manning books explain it

LLM Customization and Fine-Tuning

The student model's score divided by the teacher model's score on the same evaluation metric and set.

How it is used
Uses retention as the primary relative success metric when evaluating a distilled student against its teacher and unadapted base.
How deep it goes
Defines the score ratio and works through category-level interpretation, scaling estimates, and quality targets, while noting overlap metrics do not assess safety.
Examples
Compares a 50-example IT support evaluation with DeepSeek-R1 students on MATH-500 and AIME 2024.
What you can do after
Evaluate/understand a student’s relative quality, set a task-sensitive target, and identify when more data or a different approach may be needed.

Read how LLM Customization and Fine-Tuning explains capability retention →

Rearchitecting LLMs

Capability retention is the proportion of a model's evaluated performance preserved after a structural modification such as pruning.

How it is used
Compares preserved benchmark capability after pruning or distillation against a baseline model.
How deep it goes
Operationalizes retention through controlled benchmarks, progressive pruning curves, and configuration experiments, but does not define a universal aggregation formula.
Examples
Uses WinoGrande, Lambada, ARC-Easy, HellaSwag, PIQA, progressive block removal, and a 14-block distilled student.
What you can do after
Evaluate/understand which compressed-model configurations preserve capabilities and where further modification causes unacceptable degradation.

Read how Rearchitecting LLMs explains capability retention →

Related concepts

Questions about capability retention

What is capability retention?

Capability retention is the proportion of evaluated capabilities or performance preserved by a modified or derived model relative to a reference model.

Which Manning books explain capability retention?

LLM Customization and Fine-Tuning and Rearchitecting LLMs.

What is capability retention a kind of?

Capability retention is a kind of retention. Retention is a measure of the proportion or amount of a defined population, relationship, activity, or information that persists over a specified period.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.