Capability retention measures how much of a teacher model’s score a distilled student retains. Calculate it as student score ÷ teacher score, using the same evaluation metric and evaluation set for both. A ratio of 1.0 means the scores match; a ratio above 1.0 means the student scored higher on that evaluation. section 7.2 section 7.2.2
Compare the unadapted base model, teacher, and student—not just the student and teacher. Retention describes a relative score, not absolute quality: a student with lower retention from a stronger teacher can still be better than one with higher retention from a weaker teacher. section 7.2 section 7.7.3
IT support evaluation
On 50 held-out examples, the base model scored 0.153 Token-F1, the teacher 0.168, and the student 0.165. The student therefore retained about 98% of the teacher’s score. The teacher and student shared a base model, so this result should not be assumed for a larger capability gap. section 7.2.2 section 7.1
Use retention as a decision measure, with explicit limits:
- Set a target before viewing results, based on task criticality and a pilot of the teacher on production queries. Illustrative minimums range from 65–70% for low-stakes internal tasks to 90% or more for critical tasks. section 7.6.2
- Inspect categories as well as the overall score. Small category samples are noisy: scores above the teacher on seven prompts do not establish that the student is better. section 7.2.2
- Check the behaviors the metric misses. Token-F1 measures lexical overlap, not factual correctness, format adherence, semantic similarity, or behavioral safety; the student needs its own safety check. section 7.2.2
- If retention is too low, consider generating more teacher examples, especially for weak categories. Data volume helps, but gains diminish and depend on task complexity, teacher quality, and student capacity. section 7.2.4 section 7.7.3
Task difficulty can change the decision. A small student may retain predictable patterns while missing hard reasoning or long-tail cases; if those failures matter, consider a larger student or another technique. section 7.4.5
Table 7.4 DeepSeek-R1 distillation results by student size
|
Student size
|
MATH-500
|
Retention
|
AIME 2024
|
Retention
|
Base architecture
|
|
1.5B
|
83.9%
|
86%
|
28.9%
|
36%
|
Qwen2.5-Math-1.5B
|
|
7B
|
92.8%
|
95%
|
55.5%
|
70%
|
Qwen2.5-Math-7B
|
|
8B
|
89.1%
|
92%
|
50.4%
|
63%
|
Llama-3.1-8B
|
|
14B
|
93.9%
|
97%
|
69.7%
|
87%
|
Qwen2.5-14B
|
|
32B
|
94.3%
|
97%
|
72.6%
|
91%
|
Qwen2.5-32B
|
|
70B
|
94.5%
|
97%
|
70.0%
|
88%
|
Llama-3.3-70B-Instruct
|
Compare retention across student sizes and two reasoning benchmarks: the harder benchmark shows a much larger loss for the smallest student. section 7.4.5

