Teacher model

As explained in Build a Reasoning Model (From Scratch)

A larger model whose outputs or signals are used as training targets for a smaller student model during distillation.

A teacher model is a larger language model whose outputs supervise a smaller student during model distillation. For reasoning tasks, those outputs can include a final answer and the intermediate reasoning trace. section 8.1

The teacher supplies training targets rather than serving as the deployable model. Its generated traces let a smaller model learn reasoning behavior at lower development, inference, and hardware cost. section 8.1 section 8.9

Distillation uses teacher information in two principal ways:

• Hard distillation trains the student on teacher-generated text, treating the teacher's tokens as targets; this is usually more practical because text is cheaper to store and teacher logits may be unavailable. section 8.1 section 8.11
• Soft distillation trains the student to match the teacher's vocabulary probability distribution by minimizing KL divergence. section 8.1

Reasoning-model distillation

DeepSeek-R1, with 671 billion parameters, served as the teacher for smaller distilled variants. A practical setup used DeepSeek-R1 as teacher and Qwen3 0.6B as student, training on teacher reasoning traces and final answers from MATH problems. section 8.1 section 8.11

Figure 8.2 Hard distillation trains the student on teacher-generated tokens; soft distillation trains the student on the teacher’s full output distribution.

Compare hard distillation from teacher-generated tokens with soft distillation from the teacher's full output distribution. section 8.1

How Build a Reasoning Model (From Scratch) covers it

How deep it goes

Explains teacher outputs, reasoning traces, hard versus soft targets, and practical trade-offs, stopping short of teacher architecture internals.

Examples

Uses DeepSeek-R1 as teacher, Qwen3 0.6B as student, and MATH training problems with generated reasoning traces and answers.

What you can do after

Apply it via hard distillation by generating teacher traces and training a smaller student with supervised targets.

How it connects

Connects teacher models to model distillation, student models, reasoning traces, supervised learning, and reinforcement learning with verifiable rewards.

liveBookmanning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Building Reliable AI SystemsA larger, more capable model whose behavior or outputs serve as the target for a smaller model to learn.

Compare how 2 books explain teacher model →

Related concepts

• JSON arrayproduces · A JSON array is an ordered JSON value containing zero or more JSON values, represented as a comma-separated sequence enclosed in square brackets.
• distillationused by · Knowledge distillation is a training technique in which a student model learns to reproduce or approximate the outputs or behavior of a teacher model.
• student modelused by · A student model is a model trained to approximate or reproduce the outputs, capabilities, or behavior of another model that provides training guidance.
• no_grad()used by · no_grad() is a PyTorch context manager and decorator that disables autograd gradient tracking for enclosed tensor operations.
• synthetic datauses · Synthetic data is data generated artificially rather than collected directly from the target process, often to approximate its properties or meet specified requirements.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.