A student model is trained to reproduce behavior from a larger teacher model in a more compact form. Distillation can make a teacher’s capabilities practical to deploy without repeating the teacher’s full training process. The student’s limited capacity, however, constrains what it can absorb. section 8.2 section 8.9
In classical distillation, a frozen teacher and a trainable student process the same inputs. The student learns from the teacher’s soft probability distribution, which carries information about alternatives that a single hard label does not. A training loop can combine this signal with ordinary ground-truth labels. section 8.2 section 8.6.2
Two approaches transfer teacher behavior in different forms:
- Classical distillation matches teacher and student probability distributions. Divide both sets of logits by temperature T, compute KL divergence, and scale the result by T². Combine it with the student’s hard-label cross-entropy; update only the student. section 8.6.2
- DeepSeek-R1 distillation uses the teacher to generate reasoning traces offline, curates them with rejection sampling, and fine-tunes a pretrained student using next-token prediction. Student training needs neither the teacher’s logits nor a simultaneous teacher forward pass. section 8.5.3 section 8.1.2
Figure 8.19 Two paradigms of knowledge distillation. Classical KD (left) transfers knowledge through logit-matching with KL divergence and temperature scaling. DeepSeek’s approach (right) transfers knowledge through supervised fine-tuning on the teacher’s generated chain-of-thought text.

Compare simultaneous logit matching with offline generation of training traces. section 8.5.3
A traceable training loop
A toy teacher has four hidden layers and roughly 26,000 parameters; a two-layer student has roughly 8,800. Both accept the same input and produce scores for 10 classes. Run the teacher under torch.no_grad(), calculate the student’s hard-label and temperature-scaled soft-target losses, then backpropagate and update only the student. In a comparison using the same student architecture, the distilled student outperformed a hard-label-only baseline but did not fully match the teacher. section 8.6.1 section 8.6.2 section 8.6.3
Reasoning in smaller models
DeepSeek-R1-Distill-Qwen-1.5B fine-tunes a Qwen2.5-Math base on R1-generated data; it is not a direct transformation of the teacher’s parameters. It scores 28.9% on AIME 2024, but its small capacity also limits coding performance. On the same 32B architecture, a distilled model scores 72.6% on AIME versus 47.0% for an RL-only variant. These results do not make distillation universally superior: it transfers existing teacher behavior, while reinforcement learning can develop new behavior. section 8.1.2 section 8.9 section 7.7.6
Choose the student base for the task, not just its size. Pre-existing knowledge and architecture interact with the transferred behavior: comparable Qwen and Llama bases show different AIME results after distillation. section 8.7.3





