liveBook · Manning

Capability retention

As explained in LLM Customization and Fine-Tuning

The student model's score divided by the teacher model's score on the same evaluation metric and set.

Capability retention measures how much of a teacher model’s score a distilled student retains. Calculate it as student score ÷ teacher score, using the same evaluation metric and evaluation set for both. A ratio of 1.0 means the scores match; a ratio above 1.0 means the student scored higher on that evaluation. section 7.2 section 7.2.2

Compare the unadapted base model, teacher, and student—not just the student and teacher. Retention describes a relative score, not absolute quality: a student with lower retention from a stronger teacher can still be better than one with higher retention from a weaker teacher. section 7.2 section 7.7.3

IT support evaluation

On 50 held-out examples, the base model scored 0.153 Token-F1, the teacher 0.168, and the student 0.165. The student therefore retained about 98% of the teacher’s score. The teacher and student shared a base model, so this result should not be assumed for a larger capability gap. section 7.2.2 section 7.1

Use retention as a decision measure, with explicit limits:

  • Set a target before viewing results, based on task criticality and a pilot of the teacher on production queries. Illustrative minimums range from 65–70% for low-stakes internal tasks to 90% or more for critical tasks. section 7.6.2
  • Inspect categories as well as the overall score. Small category samples are noisy: scores above the teacher on seven prompts do not establish that the student is better. section 7.2.2
  • Check the behaviors the metric misses. Token-F1 measures lexical overlap, not factual correctness, format adherence, semantic similarity, or behavioral safety; the student needs its own safety check. section 7.2.2
  • If retention is too low, consider generating more teacher examples, especially for weak categories. Data volume helps, but gains diminish and depend on task complexity, teacher quality, and student capacity. section 7.2.4 section 7.7.3

Task difficulty can change the decision. A small student may retain predictable patterns while missing hard reasoning or long-tail cases; if those failures matter, consider a larger student or another technique. section 7.4.5

Table 7.4 DeepSeek-R1 distillation results by student size
Student size
MATH-500
Retention
AIME 2024
Retention
Base architecture
1.5B
83.9%
86%
28.9%
36%
Qwen2.5-Math-1.5B
7B
92.8%
95%
55.5%
70%
Qwen2.5-Math-7B
8B
89.1%
92%
50.4%
63%
Llama-3.1-8B
14B
93.9%
97%
69.7%
87%
Qwen2.5-14B
32B
94.3%
97%
72.6%
91%
Qwen2.5-32B
70B
94.5%
97%
70.0%
88%
Llama-3.3-70B-Instruct

Compare retention across student sizes and two reasoning benchmarks: the harder benchmark shows a much larger loss for the smallest student. section 7.4.5

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the score ratio and works through category-level interpretation, scaling estimates, and quality targets, while noting overlap metrics do not assess safety.
Examples
Compares a 50-example IT support evaluation with DeepSeek-R1 students on MATH-500 and AIME 2024.
What you can do after
Evaluate/understand a student’s relative quality, set a task-sensitive target, and identify when more data or a different approach may be needed.
How it connects
Connects retention to knowledge distillation, supervised fine-tuning, LoRA, Token-F1, and model evaluation.

Also explained in

Compare how 2 books explain capability retention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.