liveBook · Manning

Student model

As explained in Build a DeepSeek Model (From Scratch)

A distilled model is a student model trained to reproduce capabilities or behavior transferred from a larger teacher model.

A student model is trained to reproduce behavior from a larger teacher model in a more compact form. Distillation can make a teacher’s capabilities practical to deploy without repeating the teacher’s full training process. The student’s limited capacity, however, constrains what it can absorb. section 8.2 section 8.9

In classical distillation, a frozen teacher and a trainable student process the same inputs. The student learns from the teacher’s soft probability distribution, which carries information about alternatives that a single hard label does not. A training loop can combine this signal with ordinary ground-truth labels. section 8.2 section 8.6.2

Two approaches transfer teacher behavior in different forms:

  • Classical distillation matches teacher and student probability distributions. Divide both sets of logits by temperature T, compute KL divergence, and scale the result by T². Combine it with the student’s hard-label cross-entropy; update only the student. section 8.6.2
  • DeepSeek-R1 distillation uses the teacher to generate reasoning traces offline, curates them with rejection sampling, and fine-tunes a pretrained student using next-token prediction. Student training needs neither the teacher’s logits nor a simultaneous teacher forward pass. section 8.5.3 section 8.1.2
Figure 8.19 Two paradigms of knowledge distillation. Classical KD (left) transfers knowledge through logit-matching with KL divergence and temperature scaling. DeepSeek’s approach (right) transfers knowledge through supervised fine-tuning on the teacher’s generated chain-of-thought text.

Compare simultaneous logit matching with offline generation of training traces. section 8.5.3

A traceable training loop

A toy teacher has four hidden layers and roughly 26,000 parameters; a two-layer student has roughly 8,800. Both accept the same input and produce scores for 10 classes. Run the teacher under torch.no_grad(), calculate the student’s hard-label and temperature-scaled soft-target losses, then backpropagate and update only the student. In a comparison using the same student architecture, the distilled student outperformed a hard-label-only baseline but did not fully match the teacher. section 8.6.1 section 8.6.2 section 8.6.3

Reasoning in smaller models

DeepSeek-R1-Distill-Qwen-1.5B fine-tunes a Qwen2.5-Math base on R1-generated data; it is not a direct transformation of the teacher’s parameters. It scores 28.9% on AIME 2024, but its small capacity also limits coding performance. On the same 32B architecture, a distilled model scores 72.6% on AIME versus 47.0% for an RL-only variant. These results do not make distillation universally superior: it transfers existing teacher behavior, while reinforcement learning can develop new behavior. section 8.1.2 section 8.9 section 7.7.6

Choose the student base for the task, not just its size. Pre-existing knowledge and architecture interact with the transferred behavior: comparable Qwen and Llama bases show different AIME results after distillation. section 8.7.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops classical distillation through a runnable training loop and contrasts it with offline reasoning-trace fine-tuning.
Examples
Compares toy teacher and student networks and reports reasoning benchmark results for DeepSeek-R1 distilled models.
What you can do after
Build a toy knowledge-distillation pipeline from scratch in PyTorch and compare a distilled student with a hard-label baseline.
How it connects
Connects knowledge distillation with soft targets, supervised fine-tuning, rejection sampling, and reinforcement learning.

Also explained in

Compare how 6 books explain student model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.