Student model

As explained in Build a Reasoning Model (From Scratch)

A smaller model trained to imitate a teacher model's reasoning traces and answers during distillation.

A student model is a smaller language model trained to reproduce outputs from a larger teacher model. For reasoning tasks, those outputs include both intermediate reasoning traces and final answers, transferring useful behavior into a cheaper model. section 8.0 section 8.1

The workflow generates teacher traces, prepares a training dataset, trains the student with supervised learning, and evaluates checkpoints. Hard distillation uses teacher-generated tokens as targets; soft distillation matches the teacher’s vocabulary probability distribution with KL divergence. section 8.1 section 8.11

A practical student-model training run typically:

• Combines a rendered prompt with the teacher reasoning trace and final answer, optionally using <think>...</think> tags. section 8.11
• Tokenizes examples once, filters them by sequence length, and reuses the processed dataset across epochs. section 8.11
• Trains with answer-only cross-entropy, then shuffles data, backpropagates, updates weights, and monitors validation loss. section 8.11

Concrete setup

DeepSeek-R1 serves as the teacher and Qwen3 0.6B as the student. A dataset built from 12,000 MATH training problems supports the distillation run, which takes about three hours on a DGX Spark and uses roughly 15 GB of RAM. section 8.1 section 8.11

Figure 8.3 This chapter covers distillation in four main steps: (1) distillation introduction and overview; (2) dataset preparation (steps 2a-2c); (3) training via distillation (steps 3a-3c); and (4) evaluation.

Use the workflow overview to connect dataset preparation, distillation training, and evaluation. section 8.1

How Build a Reasoning Model (From Scratch) covers it

How deep it goes

Explains dataset construction, tokenization, cross-entropy training, validation, and hard-versus-soft distillation.

Examples

Uses DeepSeek-R1, Qwen3 0.6B, 12,000 MATH problems, and optional <think>...</think> reasoning tags.

What it assumes

Assumes familiarity with language-model training while teaching the teacher-student workflow and distillation choices.

What you can do after

Build it from scratch by preparing teacher traces, training a student, and evaluating checkpoints with validation loss and benchmarks.

How it connects

Connects model distillation with teacher models, reasoning traces, supervised learning, cross-entropy loss, validation loss, and RLVR.

liveBookmanning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Building Reliable AI SystemsA model trained to imitate a teacher model, often with the aim of being smaller or more efficient.

Compare how 2 books explain student model →

Related concepts

• low-rank adaptationuses · Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
• foundational modelcontrasts with · A base or foundation model is a pretrained model that provides learned parameters or capabilities as a starting point for reuse or adaptation to downstream tasks.
• distillationproduced by · Knowledge distillation is a training technique in which a student model learns to reproduce or approximate the outputs or behavior of a teacher model.
• teacher modeluses · A teacher model is a model whose outputs, labels, or learned behavior provide training signals or targets for another model.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.