A teacher model supplies supervision for training a student model, usually to transfer useful behavior into a smaller, more deployable model. The teacher may provide probability distributions, generated responses, or internal representations for the student to imitate. section 8.2 section 8.2.2 section 8.5.3
In classical knowledge distillation, a pretrained, frozen teacher and a trainable student process the same input. The student matches the teacher’s soft probability distribution, which conveys relative preferences among alternatives that a single hard label does not show. section 8.2 section 8.2.2
Two practical ways to use a teacher differ in when and what it supplies:
- Online distribution matching: run the frozen teacher on each batch, compare temperature-scaled teacher and student distributions with KL divergence, and update only the student. This requires access to teacher logits and memory for both models during training. section 8.5.3 section 8.6.2
- Offline data generation: generate and curate teacher responses first, then fine-tune the student on the resulting text with next-token prediction. The teacher is not needed during student training, and the student does not see its internal representations. section 8.5.3
A distillation training step
For a batch, run the teacher under PyTorch’s torch.no_grad() and track gradients through the student. Combine hard-label cross-entropy with KL divergence between softmax distributions at temperature T: L = α L_hard + (1−α) T² KL. Backpropagate and update only the student. With T = 3, teacher logits [5, 3, 1, −1] produce soft targets approximately [0.523, 0.268, 0.138, 0.071]. section 8.6.2
Figure 8.19 Two paradigms of knowledge distillation. Classical KD (left) transfers knowledge through logit-matching with KL divergence and temperature scaling. DeepSeek’s approach (right) transfers knowledge through supervised fine-tuning on the teacher’s generated chain-of-thought text.

Compare simultaneous probability matching with offline training on teacher-generated reasoning traces. section 8.5.3
DeepSeek-R1 as a teacher
DeepSeek-R1 generated reasoning traces for a curated 800K-example dataset used to fine-tune smaller models. On the same Qwen2.5-32B architecture, the distilled model scored 72.6% on AIME 2024 versus 47.0% for the direct-RL variant. This comparison illustrates a route to transfer existing reasoning behavior without repeating the teacher’s full RL pipeline; it does not imply that distillation reliably discovers capabilities beyond the teacher. section 7.7.6 section 8.7.2





