liveBook · Manning

Teacher model

As explained in Build a DeepSeek Model (From Scratch)

A teacher model is a model whose outputs or learned behavior provide supervision for training another model.

A teacher model supplies supervision for training a student model, usually to transfer useful behavior into a smaller, more deployable model. The teacher may provide probability distributions, generated responses, or internal representations for the student to imitate. section 8.2 section 8.2.2 section 8.5.3

In classical knowledge distillation, a pretrained, frozen teacher and a trainable student process the same input. The student matches the teacher’s soft probability distribution, which conveys relative preferences among alternatives that a single hard label does not show. section 8.2 section 8.2.2

Two practical ways to use a teacher differ in when and what it supplies:

  • Online distribution matching: run the frozen teacher on each batch, compare temperature-scaled teacher and student distributions with KL divergence, and update only the student. This requires access to teacher logits and memory for both models during training. section 8.5.3 section 8.6.2
  • Offline data generation: generate and curate teacher responses first, then fine-tune the student on the resulting text with next-token prediction. The teacher is not needed during student training, and the student does not see its internal representations. section 8.5.3

A distillation training step

For a batch, run the teacher under PyTorch’s torch.no_grad() and track gradients through the student. Combine hard-label cross-entropy with KL divergence between softmax distributions at temperature T: L = α L_hard + (1−α) T² KL. Backpropagate and update only the student. With T = 3, teacher logits [5, 3, 1, −1] produce soft targets approximately [0.523, 0.268, 0.138, 0.071]. section 8.6.2

Figure 8.19 Two paradigms of knowledge distillation. Classical KD (left) transfers knowledge through logit-matching with KL divergence and temperature scaling. DeepSeek’s approach (right) transfers knowledge through supervised fine-tuning on the teacher’s generated chain-of-thought text.

Compare simultaneous probability matching with offline training on teacher-generated reasoning traces. section 8.5.3

DeepSeek-R1 as a teacher

DeepSeek-R1 generated reasoning traces for a curated 800K-example dataset used to fine-tune smaller models. On the same Qwen2.5-32B architecture, the distilled model scored 72.6% on AIME 2024 versus 47.0% for the direct-RL variant. This comparison illustrates a route to transfer existing reasoning behavior without repeating the teacher’s full RL pipeline; it does not imply that distillation reliably discovers capabilities beyond the teacher. section 7.7.6 section 8.7.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops teacher supervision from soft-target distributions through a training loop, and contrasts online logit matching with offline response generation.
Examples
Uses a toy teacher-student network and DeepSeek-R1 reasoning traces to illustrate teacher supervision.
What you can do after
Builds a toy distillation pipeline from scratch, including a frozen teacher, student, loss, and training loop.
How it connects
Connects knowledge distillation with soft targets, KL divergence, supervised fine-tuning, rejection sampling, and reinforcement learning.

Also explained in

Compare how 6 books explain teacher model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.