Chapter 8: Distilling reasoning models for efficient reasoning

Introduction

The chapter introduces distillation as a training-time way to improve a smaller reasoning model using outputs from a larger model.

• Teacher-to-student reasoning distillation: Explain that distillation trains a smaller student model on reasoning traces and answers generated by a larger teacher model, as a training-time approach to improving reasoning performance.
  Why it matters: This idea connects the teacher-generated dataset to the student training later in the chapter.

Key terms: distillation, student model, teacher model, reasoning traces, answers

8.1 Introducing model distillation for reasoning tasks

This section introduces reasoning-model distillation, contrasts hard and soft approaches, and explains why training on teacher-generated text is the practical choice for this chapter.

• Teacher and student roles: Explain how a larger teacher model supplies outputs for training a smaller student model, and why reasoning traces accompany final answers.
  Why it matters: This is the basic relationship behind the chapter’s training workflow.
• Hard and soft training targets: Distinguish hard distillation’s teacher-generated token targets from soft distillation’s full vocabulary distributions, and describe the option that combines both.
  Why it matters: The distinction identifies what information the student is trained to reproduce.
• Practical case for hard distillation: Justify the chapter’s focus on hard distillation using the accessibility and cost of teacher outputs, and describe its motivation as an alternative to working directly with very large models or training small models with reinforcement learning alone.
  Why it matters: It explains why the chapter uses teacher-generated text rather than full teacher probabilities.

Key terms: model distillation, teacher, student, reasoning trace, hard distillation, soft distillation, supervised fine-tuning, KL divergence, output distribution, teacher logits

8.2 Generating a dataset for reasoning distillation

This section explains how to create teacher-generated training data for hard reasoning distillation and how its supervision differs from RLVR.

• Teacher-generated distillation dataset: Describe how the chapter forms a training dataset by giving non-overlapping MATH problems to DeepSeek-R1 and collecting its responses as targets for the Qwen3 0.6B student.
  Why it matters: The student needs examples of the teacher’s solutions before it can be trained by distillation.
• Distillation versus RLVR supervision: Distinguish distillation’s comparison with teacher-generated tokens from RLVR’s verifier reward based on a final answer and a reference answer.
  Why it matters: The distinction explains why collecting teacher responses replaces computing verifier rewards in this setup.
• Advance generation of teacher data: Explain why the teacher responses are generated ahead of student training and why the chapter provides a pregenerated dataset.
  Why it matters: Precomputing the teacher’s answers lets readers follow the training work without paying the generation cost themselves.

Key terms: hard distillation, student model, teacher model, MATH-500 evaluation set, training targets, RLVR, verifier, reward signal, generated tokens, target sequence

8.3 Loading the MATH training dataset for distillation

This section loads a teacher-generated MATH dataset, turns its reasoning traces and answers into distillation targets, and checks the teacher’s accuracy against ground-truth answers.

• Distillation dataset fields: Describe the fields in each dataset entry and explain why the teacher’s reasoning trace and final answer are the key supervised fine-tuning targets.
  Why it matters: Identifying the teacher-generated targets clarifies what the student learns from this dataset.
• Teacher response target format: Explain how the teacher’s reasoning trace and final answer are combined into one training target, and why the reasoning tags are optional but useful.
  Why it matters: A clear target format lets training use both parts while making the final answer easier to separate from the trace.
• Teacher answer accuracy check: Explain how the dataset’s ground-truth answers allow an accuracy check of the teacher’s generated answers, and state the reported result.
  Why it matters: Checking teacher answers puts the quality of the student’s training targets in context.

Key terms: problem, gtruth_answer, message_thinking, message_content, supervised fine-tuning, format_distilled_answer, <think>...</think> tags, reasoning trace, final answer, ground-truth labels

8.4 Building training examples

The section outlines how to turn raw data into reusable distillation training examples and explains why preprocessing them once is efficient.

• Distillation training example contents: Describe the high-level preprocessing needed to make raw dataset entries usable for training: format prompts and answers consistently, tokenize them, and store token IDs with the prompt length.
  Why it matters: This identifies what must be saved before the model can use the examples for training.
• Reusable preprocessing across epochs: Explain why distillation formats and tokenizes examples once rather than on the fly, contrasting its repeated use of examples with the sampled-and-discarded examples in RLVR. State that an epoch is one complete pass through the training dataset.
  Why it matters: Reusing prepared examples avoids repeating preprocessing each time training passes over the dataset.

Key terms: training examples, token IDs, prompt length, distillation, RLVR, training epoch, preprocessing

8.4.1 Loading and understanding the tokenizer

This section explains how to tokenize and format a prompt and teacher answer as one sequence for reasoning distillation.

• Reasoning tokenizer choice: Explain why the section uses the Qwen3 reasoning tokenizer and why using the same tokenizer variant during evaluation matters.
  Why it matters: The tokenizer choice supports reasoning tokens and keeps the evaluation setup consistent with training.
• Prompt and answer wrapping: Describe what the chat-template wrapper tokens mark and why the target answer is not wrapped separately.
  Why it matters: The model needs one coherent transition from the prompt to the teacher answer.
• Distillation training sequence: State the order of components in the token sequence used for next-token prediction.
  Why it matters: This order defines the training input from which the model predicts subsequent tokens.

Key terms: Qwen3 reasoning tokenizer, <think></think> tokens, base tokenizer, --which_model "reasoning", apply_chat_template=True, add_generation_prompt=True, chat_wrapped=False, <|im_start|>user, <|im_end|>, <|im_start|>assistant

8.4.2 Formatting and tokenizing the dataset

This subsection builds tokenized distillation examples from prompts and teacher answers, then checks a decoded example for the required format.

• Distillation example construction: Trace how build_examples renders and tokenizes a prompt, formats and tokenizes the teacher answer, concatenates them, and records the prompt length for answer-only loss computation.
  Why it matters: The prompt length identifies which tokens belong to the answer when computing the training loss.
• Decoded example format checks: Describe what decoding a training example confirms about its chat format, reasoning-trace tags, and ending token.
  Why it matters: A decoded example makes the expected training format visible for inspection.

Key terms: build_examples, prompt length, teacher answer, answer tokens, chat template, reasoning trace, <think></think>, <|im_end|>

8.4.3 Filtering and splitting the dataset

This section inspects tokenized example lengths, filters overly long examples, and splits the remaining data for training and validation.

• Tokenized sequence length inspection: Explain why the section inspects sequence lengths before filtering and what the reported lengths reveal.
  Why it matters: Inspecting lengths gives a reason for the cutoff rather than choosing one blindly.
• Maximum length filtering tradeoff: Describe the 2,048-token filter, its reported effect on the dataset, and why the section applies it.
  Why it matters: The length cap keeps the distillation workload manageable, at the cost of discarding long examples.
• Training and validation partition: Describe how the remaining examples are partitioned and distinguish the validation set’s role from the later MATH-500 evaluation.
  Why it matters: A small, reproducible validation split provides checks during training without making the loop unnecessarily slow.

Key terms: sequence lengths, outliers, tokens, 2,048 tokens, sequence length, distillation, fixed random seed, training and validation sets, MATH-500

8.5 Loading a pretrained model

This section chooses a pretrained Qwen3 base model for distillation and retains the previously prepared tokenizer that supports <think></think> tokens.

• Base model choice: Explain why distillation starts from the pretrained Qwen3 base model rather than an RL-trained model.
  Why it matters: The starting model determines how clearly you can interpret the effect of distillation.
• Distillation tokenizer choice: Explain why the tokenizer loaded alongside the base model is not used.
  Why it matters: It clarifies which tokenizer belongs in the distillation setup.

Key terms: pretrained Qwen3 base model, RL-trained model, distilled reasoning traces, teacher-generated data, base tokenizer, <think></think> token support

8.6 Computing the training and validation losses

The section defines an answer-only cross-entropy loss for distillation and explains how training and validation losses indicate whether the student is improving.

• Per Token Cross Entropy: Explain how distillation cross-entropy relates to the log probabilities of teacher-generated next tokens and why it is averaged over tokens.
  Why it matters: It explains what the loss measures and why answers of different lengths can be compared.
• Answer Only Loss Calculation: Describe which tokens contribute to the distillation loss and how the built-in cross-entropy calculation uses model predictions and targets.
  Why it matters: The student should learn to produce the teacher’s answer given the prompt, not be penalized for failing to reproduce the prompt.
• Training And Validation Losses: Distinguish training loss from validation loss and explain what their changes during training suggest.
  Why it matters: Validation loss gives a more reliable check that improvement extends beyond the examples used to update the student.

Key terms: cross-entropy, token log probabilities, answer tokens, sequence_logprob, answer-only loss, prompt, teacher’s reasoning trace, logits, targets, cross_entropy

8.7 Implementing the training loop for distillation

This section implements a distillation training loop, runs a small demonstration, and interprets its losses and saved checkpoints.

• Distillation training loop: Describe how the student model is trained across epochs and distinguish this loop from the earlier GRPO training loop.
  Why it matters: These steps explain what turns the prepared teacher-generated examples into student-model updates.
• Demonstration run settings: Explain why the example run uses a small dataset and frequent validation, and what those choices imply for a larger run.
  Why it matters: The settings keep a pipeline check from being mistaken for a full distillation experiment.
• Sequence length resource tradeoff: Describe how the maximum sequence length affects this run's GPU memory requirement and the suggested way to reduce it.
  Why it matters: This gives readers with less GPU memory a concrete way to run the experiment.
• Loss trends and checkpoints: Explain what the demonstration's training and validation losses indicate and why its checkpoints are useful.
  Why it matters: The observed losses help check whether the short run is learning, while checkpoints preserve models for later work.

Key terms: epoch, cross-entropy distillation loss, GRPO loss, optimizer step, validation set, log_every, validation loss, distillation pipeline, maximum sequence length, GPU memory

8.8 Evaluating the distilled model

This section evaluates a distilled reasoning model through validation-loss trends and MATH-500 checkpoint results, then interprets the performance and cost tradeoffs.

• Training and validation loss interpretation: Distinguish how training and validation losses are computed in this run, and explain why the validation curve is more useful for judging progress. Describe what its initial drop and later flattening indicate.
  Why it matters: The two curves do not provide equally clear evidence of training progress.
• Checkpoint benchmark evaluation: Describe how saved distilled checkpoints are evaluated on MATH-500 and what the DeepSeek-R1-run results show across epochs.
  Why it matters: Benchmarking checkpoints shows whether lower validation loss accompanies better task accuracy.
• Teacher alignment and distillation tradeoff: Explain the proposed reason the Qwen3-teacher run performs better here, and describe the performance-versus-cost tradeoff illustrated by the distilled student.
  Why it matters: The results show both why teacher choice may matter and what efficiency distillation buys.

Key terms: training loss, validation loss, held-out set, diminishing returns, saved checkpoints, MATH-500, reasoning tokenizer, verifier-based utilities, teacher, student

8.9 Future directions for reasoning models

The section identifies likely directions for reasoning models: stronger teachers and distillation, adaptive inference effort, richer training rewards, and agent applications.

• Teacher and student strategy: Describe the continuing strategy of improving a reasoning teacher, collecting high-quality reasoning traces, and distilling them into smaller students. Explain how the section pairs RLVR for flagship models with distillation for computationally efficient models.
  Why it matters: This connects improvements in powerful reasoning models to models that are more practical to deploy.
• Adaptive inference effort: Explain why reasoning effort and answer length should vary by task, and describe the application layer's role in choosing a short answer, more reasoning, or an early stop.
  Why it matters: Allocating reasoning effort to the task can avoid unnecessary long responses while leaving room for harder problems.
• Intermediate step rewards: Distinguish rewards based on final answers from process rewards that check intermediate reasoning steps, and explain why the latter may help reasoning reliability.
  Why it matters: A correct final answer alone may give less feedback about the reasoning that produced it.
• Agent tasks and rewards: Describe how reasoning models' role in larger agent applications expands the behaviors they must perform, and explain why this motivates rewards beyond a single correctness score.
  Why it matters: An agent's success involves more than getting an isolated math or coding answer right.

Key terms: reasoning teacher, reasoning traces, student models, RLVR, distillation, inference scaling, application layer, reasoning budget, reasoning effort, final answers

8.10 Conclusions

Closes the book’s main technical material and introduces forthcoming pointers for further learning.

8.10.1 What’s next

The section recommends combining and comparing the book’s methods to find what helps most in a given setting, and points readers to resources for further work.

• Combining and comparing reasoning methods: Describe a possible sequence of distilling a smaller model, continuing its training with RLVR, and applying inference-time scaling at deployment. Explain why comparing such combinations can be useful.
  Why it matters: Comparisons can guide your choice of methods for the setting you are working in.

Key terms: distill, RLVR, inference-time scaling, self-consistency, self-refinement

8.10.2 Staying up to date in a fast-moving field

The section explains how to stay current with reasoning-model research by interpreting new work through durable fundamentals and filtering a fast-moving stream of sources.

• Durable reasoning model fundamentals: Explain why the book’s core ideas remain useful as particular algorithms and datasets change, and how to use those ideas to interpret new methods.
  Why it matters: These fundamentals give you a way to make sense of new techniques without treating each one as entirely new.
• Selective research monitoring: Describe how to use papers, model reports, practitioner discussions, and AI tools to follow new developments without trying to read everything.
  Why it matters: A selective approach helps you notice useful developments while checking important claims against their original sources.

Key terms: model evaluation, inference-time, training-time, training losses, reward signals, arXiv, technical summaries, model cards, practitioner discussions, deep research tools

8.11 Summary

The section summarizes how teacher-generated reasoning examples are prepared, used to train a smaller student model, and monitored during distillation.