Chapter 8: Knowledge distillation: Making powerful models practical
The introduction motivates knowledge distillation as a way to make DeepSeek-R1’s reasoning ability practical to deploy in much smaller models.
After this chapter you will be able to
- Practical motivation for distillation. Explain why the full DeepSeek-R1 is impractical for personal devices and how distillation addresses that problem. Cite the reported 1.5-billion-parameter model as an example of the potential payoff.
- Deployment as a separate challenge. Explain why building a powerful model is not the end of the task: it must also be deployed so users can access it.
- MoE memory requirement. Explain why activating only a fraction of DeepSeek-R1’s parameters per token does not make the full model fit on a single GPU.
- Real-world deployment gap. Describe how hardware, per-response costs, and cloud-service constraints keep the full model out of practical settings such as local or offline tools.
- Practical reasoning compression. Explain how the 1.5B distilled model illustrates a reduction in model size and memory requirements, and qualify its reported math performance rather than treating it as an across-the-board win.
- Behavior transfer into existing models. Describe what DeepSeek-R1 contributes to the distilled models and distinguish that from directly shrinking the teacher’s parameters.
- Teacher and student roles. Explain how knowledge distillation uses an already-trained, larger teacher model to train a smaller student to reproduce its behavior rather than learning only from raw data.
- Frozen teacher and soft targets. Describe how the frozen teacher and student process the same training data, and distinguish the teacher’s full output probability distribution from one-hot ground-truth labels.
- Soft Labels and Dark Knowledge. Explain how a teacher’s probability distribution conveys the correct answer and the relative plausibility of incorrect classes, unlike a one-hot hard label. Describe this information about class relationships as dark knowledge and why it helps a student learn.
- Soft Targets and Generalization. Explain how the master chef analogy contrasts hard labels with a teacher’s probability distribution, and why learning from the distribution can improve generalization.
- Temperature and dark knowledge. Explain why the teacher’s output at temperature T=1 can obscure useful information, and describe how temperature scaling is applied to its logits.
- Temperature and probability softness. Explain how the temperature parameter changes a softmax distribution, including what happens at T = 1, at moderately higher values, and as T approaches infinity.
- Temperature Effects on Probabilities. Trace how dividing the teacher’s logits by temperature before softmax changes the four-class distribution. Contrast the sharply peaked result at T=1, the informative spread at T=3, and the nearly uniform result at T=10.
- Class similarity in soft targets. Explain how the teacher’s soft targets convey a similarity hierarchy and why that can help a student learn from fewer examples than hard labels alone.
- Temperature squared loss scaling. Explain why increasing temperature weakens the gradient from the teacher–student KL loss and why multiplying the soft-target loss by T² compensates for that weakening.
- Temperature signal trade-off. Explain why increasing temperature helps only up to a point: it softens a peaked distribution to reveal inter-class relationships, but excessive softening weakens the useful signal.
- Student capacity and temperature. Explain why the preferred temperature depends on student size, and describe the section’s suggested temperature ranges for small, medium, and large students.
- Hard-label training limitations. Explain how a student trains from scratch on hard labels and why this makes learning relationships between classes slower than learning from a teacher’s soft targets.
- Peaked teacher targets at T=1. Explain why using the teacher’s output probabilities at T=1 improves little on hard-label training, and how raising the temperature addresses the problem.
- Temperature-scaled class relationships. Explain how raising the teacher’s softmax temperature makes less likely classes more visible to the student without changing their ranking.
- Complementary training targets. Explain why the student needs both hard labels and the teacher’s temperature-scaled soft targets.
- Combined distillation objective. Explain how the total loss combines hard-label cross-entropy with temperature-scaled, T²-weighted soft-target KL divergence, and what each term contributes.
- Temperature scaling and loss weights. Explain why the soft loss is multiplied by T² and how α and β balance ground truth against teacher guidance.
- Classification and language output differences. Distinguish the fixed probability distribution over shared classes used in classical knowledge distillation from an LLM’s token-by-token generation of potentially long reasoning chains.
- Reasoning Traces as Teacher Knowledge. Explain why the valuable output of a reasoning teacher is its chain-of-thought text rather than its per-token probabilities. Describe how a trace can show a strategy for breaking a problem into steps, applying techniques, and verifying an answer.
- Text-Based Student Fine-Tuning. Describe how chain-of-thought distillation trains the student on complete teacher-generated reasoning traces using standard supervised next-token prediction, instead of matching teacher logits with KL divergence and temperature scaling.
- Rejection sampling for distillation. Explain how DeepSeek-R1 generated multiple responses per prompt and kept only responses that passed correctness and formatting checks, rather than using the first response.
- Reasoning and general data balance. Explain the approximate reasoning-versus-non-reasoning composition of the 800K-sample dataset and why general-purpose examples were included.
- Two distillation paradigms. Distinguish how classical KD trains a student from how DeepSeek trains one: matching the teacher’s softened output probabilities versus learning from curated teacher-generated reasoning traces with next-token prediction.
- Offline teacher student separation. Explain why DeepSeek does not need the teacher loaded during student training, and contrast that with classical KD’s real-time teacher predictions.
- Distilled model families and bases. Describe the six distilled models across the Qwen2.5 and Llama-3 families, and explain why their starting bases matter for reasoning performance.
- Reusable distillation data economics. Explain how paying once to generate teacher responses makes distillation economical across multiple student models, and compare that data-generation cost with the teacher’s RL training cost.
- Teacher and student architectures. Describe how the toy teacher and student differ in depth, width, and parameter count while using the same embedding dimensions and output space. Explain that the smaller student aims to approximate the larger teacher’s behavior.
- Frozen teacher and trainable student. Trace one distillation training step, explaining why the teacher runs without gradient tracking while the student receives the update.
- Combined hard and soft losses. Describe how the student's hard-label cross-entropy and its temperature-scaled agreement with the teacher form the weighted training loss, and explain what the soft term adds.
- Three-way student comparison. Describe the baseline, distilled student, and teacher in the comparison, and explain their relative performance.
- Small model benchmark upset. Explain how the 1.5B distilled model compared with GPT-4o and Claude-3.5-Sonnet on AIME 2024, and why its size makes the result striking.
- Broader distilled math results. Describe how the 32B and 7B distilled models performed against o1-mini, including what the MATH-500 results suggest about distillation.
- Distillation performance boundaries. Distinguish the small model’s strong math performance from its weaker coding result, and compare the best distilled model with its teacher on AIME.
- Same-base training comparison. Describe how DeepSeek compared direct RL with distillation on the Qwen2.5-32B base, and state the main performance result.
- RL convergence and capacity ceiling. Explain why the section treats the similar results of two independent RL-trained 32B models as evidence for a capacity-related plateau, and how distillation differs.
- Distilled Model Scaling. Describe the steady improvement in AIME 2024 performance as distilled models grow, while recognizing that size alone does not determine the result.
- Base Model Selection. Explain how a student's pre-existing capabilities affect what it gains from distillation and how that shapes base-model choice.
- Exploration Versus Imitation. Explain why the small models discussed struggled with direct RL and how distillation lets them acquire reasoning behavior instead.
- Capacity Gap Across Tasks. Explain how a student’s limited capacity can cause it to lose some of a much larger teacher’s knowledge, producing uneven performance across tasks. Use the 1.5B model’s math and coding results to illustrate why structured mathematical reasoning compresses better than complex coding.
- Teacher Knowledge Boundary. Explain why distillation cannot transfer a solution the teacher does not have, including the effect of retaining only correct teacher responses during rejection sampling.
- Discovery And Deployment Roles. Describe the complementary roles assigned to reinforcement learning and distillation, and why neither replaces the other.
- Distillation reproducibility gap. Explain why releasing the distilled models and some training settings does not make DeepSeek’s results exactly reproducible, while still allowing others to train competitive models.
- On-policy distillation trade-off. Distinguish static, off-policy teacher data from on-policy distillation, in which a student generates attempts that the teacher corrects or scores during training. Explain why this can target the student's own weaknesses but costs more.
- Distillation as deployment bridge. Explain the section's concluding lesson that distillation transfers knowledge from a massive model into much smaller models, connecting frontier research to practical deployment rather than treating scale alone as the measure of impact.
What it builds on
- Distillation into smaller models (chapter 7) · Introduces distillation into smaller models.
- The sparsity solution: Activating only a subset of experts per token (chapter 4) · Introduces activating only a subset of experts per token.
- The dark side of the KV cache: The memory cost (chapter 2) · Introduces the memory cost of the KV cache.
- What is quantization? (chapter 5) · Introduces quantization, one of the alternative approaches named here.
- Rejection sampling as data construction (chapter 7) · Introduces rejection sampling as data construction.
- Designing a verifier (chapter 7) · Introduces the idea of checking generated responses with a verifier.
- GRPO: DeepSeek's value model simplification (chapter 7) · GRPO as a reinforcement-learning approach.
Section by section
Introduction
The introduction motivates knowledge distillation as a way to make DeepSeek-R1’s reasoning ability practical to deploy in much smaller models.
- Practical motivation for distillation: Explain why the full DeepSeek-R1 is impractical for personal devices and how distillation addresses that problem. Cite the reported 1.5-billion-parameter model as an example of the potential payoff.
Why it matters: It connects the chapter’s technique to the practical problem of deploying reasoning models with limited hardware.
Key terms: DeepSeek-R1, knowledge distillation, distilled model
8.1 Why 671 billion parameters won’t fit in your pocket
The section frames deployment as a challenge distinct from building a powerful language model.
- Deployment as a separate challenge: Explain why building a powerful model is not the end of the task: it must also be deployed so users can access it.
Why it matters: A powerful model has practical value only if people can use it.
Key terms: deploying
8.1.1 The cost of intelligence
This section quantifies why the full DeepSeek-R1 model is costly to run and explains the gap between its reasoning capability and practical deployment.
- MoE memory requirement: Explain why activating only a fraction of DeepSeek-R1’s parameters per token does not make the full model fit on a single GPU.
Why it matters: Sparse activation reduces work per token, but it does not remove the need to store the model. - Real-world deployment gap: Describe how hardware, per-response costs, and cloud-service constraints keep the full model out of practical settings such as local or offline tools.
Why it matters: A model’s reasoning capability has limited reach if the people who need it cannot afford or access its deployment setup. - Compression approaches and distillation: Distinguish the compression approaches named here and state why the section points toward knowledge distillation.
Why it matters: It connects the deployment problem to the chapter’s choice to examine distillation.
Key terms: Mixture-of-Experts, active parameters, model weights, FP16, inference, KV Cache, deployment gap, cloud API, quantization, pruning
8.1.2 The dream: Reasoning in your pocket
This section shows that a small distilled model can make some of DeepSeek-R1’s reasoning capabilities practical to run, while clarifying what is transferred and where the smaller model falls short.
- Practical reasoning compression: Explain how the 1.5B distilled model illustrates a reduction in model size and memory requirements, and qualify its reported math performance rather than treating it as an across-the-board win.
Why it matters: The striking size reduction matters because it puts useful reasoning on much smaller hardware, but the benchmark result has a specific scope. - Behavior transfer into existing models: Describe what DeepSeek-R1 contributes to the distilled models and distinguish that from directly shrinking the teacher’s parameters.
Why it matters: This distinction prevents the model-size comparison from being mistaken for a procedure that physically shrinks R1’s weights.
Key terms: 671B model, DeepSeek-R1-Distill-Qwen-1.5B, AIME 2024, coding tasks, distilled models, Qwen2.5, Llama-3, 800K curated reasoning examples, fine-tune
8.2 The teacher-student paradigm
This section introduces knowledge distillation as training a small student model to reproduce a larger teacher model’s behavior using the teacher’s output probabilities.
- Teacher and student roles: Explain how knowledge distillation uses an already-trained, larger teacher model to train a smaller student to reproduce its behavior rather than learning only from raw data.
Why it matters: This is the purpose behind the teacher–student training setup. - Frozen teacher and soft targets: Describe how the frozen teacher and student process the same training data, and distinguish the teacher’s full output probability distribution from one-hot ground-truth labels.
Why it matters: The teacher’s probabilities are the training signal that distinguishes this setup from training on hard labels.
Key terms: knowledge distillation, teacher, student, model compression, frozen teacher, soft targets, hard labels, full probability distribution
8.2.1 Why soft labels carry more information than hard labels
Explains why a teacher’s soft probability distribution gives a student more useful information than a hard label.
- Soft Labels and Dark Knowledge: Explain how a teacher’s probability distribution conveys the correct answer and the relative plausibility of incorrect classes, unlike a one-hot hard label. Describe this information about class relationships as dark knowledge and why it helps a student learn.
Why it matters: The student can learn from the teacher’s judgments about near misses, not just whether an answer is correct.
Key terms: hard labels, one-hot vector, soft labels, dark knowledge, inter-class similarity relationships
8.2.2 The master chef analogy
This section uses a cooking analogy to explain why a teacher’s soft targets can help a student generalize beyond what hard labels teach, then illustrates distillation’s practical value and its earlier scope.
- Soft Targets and Generalization: Explain how the master chef analogy contrasts hard labels with a teacher’s probability distribution, and why learning from the distribution can improve generalization.
Why it matters: The analogy explains what the student gains from a teacher’s soft targets beyond a correct answer. - DistilBERT Compression Example: Describe how DistilBERT illustrates the practical value of distillation, including what it matched and the reported compression and performance results.
Why it matters: DistilBERT gives a concrete example of distillation making a capable model cheaper to deploy. - Earlier Distillation Task Scope: Distinguish the earlier distillation successes described here from the reasoning transfer pursued by DeepSeek-R1.
Why it matters: This distinction keeps the earlier evidence for distillation from being mistaken for evidence that reasoning transfer works the same way.
Key terms: hard label, soft targets, teacher’s probability distribution, generalization, DistilBERT, soft-target distillation, intermediate layer matching, hidden states, fixed set of output classes, reasoning
8.3 Temperature and dark knowledge
This section explains why a peaked teacher output can hide useful relationships between incorrect classes and introduces temperature scaling as a way to reveal them.
- Temperature and dark knowledge: Explain why the teacher’s output at temperature T=1 can obscure useful information, and describe how temperature scaling is applied to its logits.
Why it matters: It explains what information distillation seeks beyond the teacher’s most likely answer and how the section proposes making that information visible.
Key terms: incorrect classes, dark knowledge, logits, temperature scaling, softmax, T=1
8.3.1 The temperature-scaled softmax
This subsection explains how temperature changes softmax probabilities and why moderate softening can reveal useful information for distillation.
- Temperature and probability softness: Explain how the temperature parameter changes a softmax distribution, including what happens at T = 1, at moderately higher values, and as T approaches infinity.
Why it matters: Choosing a temperature involves revealing the teacher’s relative preferences without flattening them away.
Key terms: temperature, temperature-scaled softmax, softness, inter-class similarity structure
8.3.2 A numerical walkthrough: seeing the dark knowledge emerge
A four-class example shows how temperature changes a teacher’s softmax probabilities and why a moderately softened distribution reveals useful class relationships.
- Temperature Effects on Probabilities: Trace how dividing the teacher’s logits by temperature before softmax changes the four-class distribution. Contrast the sharply peaked result at T=1, the informative spread at T=3, and the nearly uniform result at T=10.
Why it matters: The example shows why a teacher’s most likely class alone is less informative than a suitably softened distribution. - Useful Versus Maximum Entropy: Explain why the example favors useful entropy rather than maximum entropy: the distribution should reveal the ranking of alternative classes without becoming nearly uniform.
Why it matters: It explains why the most uniform teacher output is not the best teaching signal.
Key terms: logits, temperature, softmax, dark knowledge, entropy, useful entropy, Goldilocks zone
8.3.3 What the dark knowledge reveals
This section explains how temperature-softened teacher probabilities reveal relationships between classes and give a student richer supervision than hard labels.
- Class similarity in soft targets: Explain how the teacher’s soft targets convey a similarity hierarchy and why that can help a student learn from fewer examples than hard labels alone.
Why it matters: It explains why learning from a teacher can be more data-efficient than learning only from correct-answer labels.
Key terms: dark knowledge, soft targets, hard labels, inter-class similarity structure
8.3.4 Why T² scaling matters
Explains why temperature-scaled distillation multiplies the soft-target loss by T² to keep its learning signal strong.
- Temperature squared loss scaling: Explain why increasing temperature weakens the gradient from the teacher–student KL loss and why multiplying the soft-target loss by T² compensates for that weakening.
Why it matters: Without this scaling, raising the temperature to reveal more of the teacher’s dark knowledge would also make the soft-target loss much weaker.
Key terms: T² scaling factor, KL loss, soft-target loss, temperature, dark knowledge
8.3.5 The temperature Goldilocks zone
Explains how to choose a distillation temperature that reveals useful dark knowledge without overwhelming the student model.
- Temperature signal trade-off: Explain why increasing temperature helps only up to a point: it softens a peaked distribution to reveal inter-class relationships, but excessive softening weakens the useful signal.
Why it matters: The student needs visible relationships between classes, not a distribution so flat that those relationships lose their signal. - Student capacity and temperature: Explain why the preferred temperature depends on student size, and describe the section’s suggested temperature ranges for small, medium, and large students.
Why it matters: Choosing temperature to match student capacity avoids asking a small model to learn more subtlety than it can retain.
Key terms: temperature, Goldilocks zone, dark knowledge, distribution, student, capacity
8.4 Building the distillation loss: From naive to powerful
Introduces a progression of three approaches that will build toward the complete knowledge distillation loss.
8.4.1 Attempt #1: Training from scratch with hard labels
The section explains why training a student from scratch on hard labels is a slow baseline compared with learning from a teacher’s soft targets.
- Hard-label training limitations: Explain how a student trains from scratch on hard labels and why this makes learning relationships between classes slower than learning from a teacher’s soft targets.
Why it matters: This explains what teacher guidance adds beyond simply giving the student more labeled examples.
Key terms: student model, hard labels, one-hot label, soft targets, inter-class relationships
8.4.2 Attempt #2: Matching the teacher’s output at T=1
This section explains why training a student on the teacher’s output probabilities at T=1 yields only a small improvement over hard labels.
- Peaked teacher targets at T=1: Explain why using the teacher’s output probabilities at T=1 improves little on hard-label training, and how raising the temperature addresses the problem.
Why it matters: It explains why teacher probabilities alone do not necessarily pass the teacher’s nuanced class relationships to the student.
Key terms: teacher’s output probabilities, T=1, hard label, gradients, inter-class structure, temperature
8.4.3 Temperature-scaled soft targets
This section shows how raising softmax temperature makes the teacher’s soft targets more informative and explains why distillation also needs hard labels.
- Temperature-scaled class relationships: Explain how raising the teacher’s softmax temperature makes less likely classes more visible to the student without changing their ranking.
Why it matters: The student gets a learning signal about plausible alternatives, not just the teacher’s top choice. - Complementary training targets: Explain why the student needs both hard labels and the teacher’s temperature-scaled soft targets.
Why it matters: Using both targets preserves correctness while passing on what the teacher reveals about other possible answers.
Key terms: temperature, soft targets, dark knowledge, inter-class relationships, hard labels, grounding
8.4.4 The complete distillation loss
This section combines hard-label and teacher-guided losses into a classical distillation objective and explains how it trains only the student.
- Combined distillation objective: Explain how the total loss combines hard-label cross-entropy with temperature-scaled, T²-weighted soft-target KL divergence, and what each term contributes.
Why it matters: The two terms let a small model learn from the teacher without losing sight of correct answers. - Temperature scaling and loss weights: Explain why the soft loss is multiplied by T² and how α and β balance ground truth against teacher guidance.
Why it matters: These choices determine whether the teacher’s signal remains effective without displacing the ground truth. - Frozen teacher student updates: Describe which model receives gradients from the combined loss and what changes a standard training pipeline needs.
Why it matters: The teacher supplies a stable training target while the smaller student learns from it.
Key terms: hard-label loss, soft-target loss, cross-entropy, KL divergence, dark knowledge, T² scaling, α, β, gradient magnitude, teacher model
8.5 DeepSeek-R1’s distillation recipe
This section contrasts fixed-class knowledge distillation with language generation to frame the challenge of distilling reasoning.
- Classification and language output differences: Distinguish the fixed probability distribution over shared classes used in classical knowledge distillation from an LLM’s token-by-token generation of potentially long reasoning chains.
Why it matters: This distinction explains why the classification setup does not, by itself, answer how to distill reasoning.
Key terms: classical knowledge distillation, output space, probability distribution, token by token, chains of reasoning
8.5.1 From logits to language: a paradigm shift
This section explains why DeepSeek-R1 distillation transfers generated reasoning traces rather than matching the teacher’s token probabilities.
- Reasoning Traces as Teacher Knowledge: Explain why the valuable output of a reasoning teacher is its chain-of-thought text rather than its per-token probabilities. Describe how a trace can show a strategy for breaking a problem into steps, applying techniques, and verifying an answer.
Why it matters: It identifies what the student is meant to gain from a reasoning teacher, beyond a final answer. - Text-Based Student Fine-Tuning: Describe how chain-of-thought distillation trains the student on complete teacher-generated reasoning traces using standard supervised next-token prediction, instead of matching teacher logits with KL divergence and temperature scaling.
Why it matters: It distinguishes this distillation recipe from the classical approach described earlier in the chapter.
Key terms: chain-of-thought (CoT) reasoning traces, per-token probability distribution, coherent reasoning strategy, chain-of-thought distillation, supervised fine-tuning, next-token prediction, logit distribution, KL divergence, temperature scaling
8.5.2 The 800K training dataset: Rejection sampling at scale
This section explains how DeepSeek curated roughly 800K teacher-generated examples for distillation, balancing verified reasoning traces with general-purpose responses.
- Rejection sampling for distillation: Explain how DeepSeek-R1 generated multiple responses per prompt and kept only responses that passed correctness and formatting checks, rather than using the first response.
Why it matters: The student trains on the selected examples, so filtering determines what it is taught to imitate. - Domain-specific correctness checks: Describe how DeepSeek checked math and code responses, and how it judged candidates in domains less amenable to direct checking.
Why it matters: Different tasks require different ways to decide which teacher responses are trustworthy. - Reasoning and general data balance: Explain the approximate reasoning-versus-non-reasoning composition of the 800K-sample dataset and why general-purpose examples were included.
Why it matters: The mix aims to transfer reasoning without turning the smaller model into a narrow specialist. - Task-appropriate response styles: Describe why detailed reasoning traces and concise responses to simple queries both appear in the curated data.
Why it matters: A student can copy both the reasoning patterns and the presentation habits in its training examples.
Key terms: rejection sampling, candidate responses, distillation dataset, ground truth, DeepSeek-V3, generative reward model, reasoning samples, non-reasoning samples, illustrative estimate, reasoning traces
8.5.3 Classical KD versus DeepSeek’s approach
This section contrasts classical logit-matching distillation with DeepSeek’s offline, text-based approach and explains why the latter simplifies student training.
- Two distillation paradigms: Distinguish how classical KD trains a student from how DeepSeek trains one: matching the teacher’s softened output probabilities versus learning from curated teacher-generated reasoning traces with next-token prediction.
Why it matters: The two approaches pass on the teacher’s behavior through different training signals. - Offline teacher student separation: Explain why DeepSeek does not need the teacher loaded during student training, and contrast that with classical KD’s real-time teacher predictions.
Why it matters: Separating data generation from student training avoids having to load the 671B teacher alongside the student. - SFT without reinforcement learning: State that DeepSeek’s distilled models use SFT without an RL stage, and explain what the section takes their reported performance to show.
Why it matters: It clarifies that the reported distilled-model results are attributed here to curated data and SFT, not to a student RL stage.
Key terms: classical KD, logits, KL divergence, temperature scaling, T² correction, rejection sampling, soft targets, offline, static dataset, SFT
8.5.4 The six distilled models
This section identifies DeepSeek-R1’s six distilled models and explains how base-model choice and fine-tuning settings affect the resulting students.
- Distilled model families and bases: Describe the six distilled models across the Qwen2.5 and Llama-3 families, and explain why their starting bases matter for reasoning performance.
Why it matters: A distilled model’s performance reflects both what distillation teaches it and what its base model could already do. - Fine-tuning configuration and learning rates: State the shared fine-tuning settings and explain why learning rates were adjusted by model size.
Why it matters: The setup shows how the same dataset was used across students while optimization was adjusted to their sizes.
Key terms: Qwen2.5, Llama 3, Qwen2.5-Math, math-specialized base, general-purpose bases, 800K dataset, cosine learning rate decay, maximum context length, learning rates
8.5.5 The economics of distillation
This section explains why generating a reusable reasoning dataset from a costly teacher can be cheaper than training each small model independently with reinforcement learning.
- Reusable distillation data economics: Explain how paying once to generate teacher responses makes distillation economical across multiple student models, and compare that data-generation cost with the teacher’s RL training cost.
Why it matters: Reusing teacher-generated data spreads its creation cost across many smaller models.
Key terms: rejection sampling, distillation data, student models, standard SFT
8.6 Implementing knowledge distillation in PyTorch
Introduces a planned PyTorch knowledge distillation pipeline using small, traceable teacher and student models.
8.6.1 Defining teacher and student models
Defines toy teacher and student models that share an input and output setup but differ substantially in capacity.
- Teacher and student architectures: Describe how the toy teacher and student differ in depth, width, and parameter count while using the same embedding dimensions and output space. Explain that the smaller student aims to approximate the larger teacher’s behavior.
Why it matters: This size contrast sets up the distillation task: transferring behavior from a larger model to a smaller one.
Key terms: teacher, student, embedding layer, output space, representational capacity
8.6.2 The training loop
This section walks through a distillation training step in which a frozen teacher and trainable student process the same inputs, and a combined loss updates only the student.
- Frozen teacher and trainable student: Trace one distillation training step, explaining why the teacher runs without gradient tracking while the student receives the update.
Why it matters: The teacher supplies a target without incurring the cost of computing gradients for parameters that will not change. - Combined hard and soft losses: Describe how the student's hard-label cross-entropy and its temperature-scaled agreement with the teacher form the weighted training loss, and explain what the soft term adds.
Why it matters: The combined loss trains the student against the correct label while also drawing on the teacher's relationships among classes.
Key terms: frozen teacher model, student model, torch.no_grad(), backpropagation, hard-label loss, soft-target loss, cross-entropy, KL divergence, temperature, T²
8.6.3 Comparing the three approaches
Compares three ways to evaluate a student model and explains where distillation provides the greatest benefit.
- Three-way student comparison: Describe the baseline, distilled student, and teacher in the comparison, and explain their relative performance.
Why it matters: This comparison isolates the practical benefit of teacher guidance for the student. - Benefit on ambiguous examples: Explain why the reported distillation advantage is larger on ambiguous examples than on easy ones, and relate that pattern to the benchmark comparison mentioned here.
Why it matters: It shows why the gain from distillation is not equally visible on every test example.
Key terms: baseline, hard labels, soft targets, T=3, teacher (upper bound), dark knowledge, ambiguous, borderline examples, AIME
8.7 The payoff: An empirical head-to-head comparison
Introduces the upcoming empirical comparison of DeepSeek-R1’s distilled models without presenting the results.
8.7.1 The headline: 1.5B beats GPT-4o
This section compares DeepSeek-R1’s distilled models with frontier models on reasoning benchmarks, highlighting strong math results alongside limits in coding and teacher-level performance.
- Small model benchmark upset: Explain how the 1.5B distilled model compared with GPT-4o and Claude-3.5-Sonnet on AIME 2024, and why its size makes the result striking.
Why it matters: The size-to-performance contrast is the section’s headline evidence for the practical value of distillation. - Broader distilled math results: Describe how the 32B and 7B distilled models performed against o1-mini, including what the MATH-500 results suggest about distillation.
Why it matters: These comparisons show that the headline result is part of a broader pattern, especially on math benchmarks. - Distillation performance boundaries: Distinguish the small model’s strong math performance from its weaker coding result, and compare the best distilled model with its teacher on AIME.
Why it matters: The benchmark wins should not be mistaken for equal ability across tasks or full preservation of the teacher’s performance. - Base architecture comparison: Describe the AIME difference between similarly sized Qwen- and Llama-based distilled models and the section’s tentative explanation.
Why it matters: Model size alone does not explain the differences among distilled models in this comparison.
Key terms: DeepSeek-R1-Distill-Qwen-1.5B, AIME 2024, GPT-4o, Claude-3.5-Sonnet, Qwen distilled model, o1-mini, MATH-500, AIME, GPQA Diamond, LiveCodeBench
8.7.2 The critical comparison: Distillation versus direct RL
This section compares distillation with direct reinforcement learning on the same 32B base model and explains what the results imply for building capable smaller models.
- Same-base training comparison: Describe how DeepSeek compared direct RL with distillation on the Qwen2.5-32B base, and state the main performance result.
Why it matters: Using the same base model makes the training approach the central difference in this comparison. - RL convergence and capacity ceiling: Explain why the section treats the similar results of two independent RL-trained 32B models as evidence for a capacity-related plateau, and how distillation differs.
Why it matters: The independent RL reference helps explain why the result is more than a comparison with one unsuccessful training run. - Distillation’s practical boundary: Distinguish the section’s conclusion that distillation is an economical way to make capable small models from its limit on creating new knowledge.
Why it matters: The result favors distillation for practical small models without making it a substitute for advancing the knowledge frontier.
Key terms: Qwen2.5-32B, DS-R1-Zero-32B, DS-R1-Distill-32B, SFT, AIME 2024, QwQ-32B, capacity, RL-from-scratch, distillation, large-scale reinforcement learning
8.7.3 Scaling behavior
This section shows how distilled reasoning performance changes with student size and base model, and why distillation succeeds where direct reinforcement learning with small models struggled.
- Distilled Model Scaling: Describe the steady improvement in AIME 2024 performance as distilled models grow, while recognizing that size alone does not determine the result.
Why it matters: Model size is useful for anticipating gains, but it is not enough to choose a student model. - Base Model Selection: Explain how a student's pre-existing capabilities affect what it gains from distillation and how that shapes base-model choice.
Why it matters: Choosing a base that already has relevant strengths can matter as much as adding parameters. - Exploration Versus Imitation: Explain why the small models discussed struggled with direct RL and how distillation lets them acquire reasoning behavior instead.
Why it matters: It explains why teacher-generated reasoning data can help a small model when direct RL does not.
Key terms: smooth scaling, model size, AIME 2024, base model, pre-existing knowledge, Qwen2.5-Math, Llama, direct RL, exploration, imitation
8.8 The limits of distillation
8.8.1 The capacity gap
This section explains why a small distilled student can retain the teacher’s abilities unevenly across tasks and uses that capacity gap to guide student-model size choices.
- Capacity Gap Across Tasks: Explain how a student’s limited capacity can cause it to lose some of a much larger teacher’s knowledge, producing uneven performance across tasks. Use the 1.5B model’s math and coding results to illustrate why structured mathematical reasoning compresses better than complex coding.
Why it matters: It explains why a strong distilled result in one domain does not guarantee equally strong performance in another. - Student Size Selection Guidelines: Describe the section’s practical student-size ranges for narrow reasoning, general-purpose reasoning, and near-teacher quality, including their deployment trade-offs.
Why it matters: These ranges help match a distilled model’s intended tasks to realistic deployment hardware.
Key terms: capacity gap, student model, teacher, MATH-500, LiveCodeBench, 1.5B–3B, 7B–14B, 32B–70B, domain-specific, general-purpose reasoning
8.8.2 The knowledge frontier
This section explains why distillation is bounded by the teacher’s knowledge and how it complements reinforcement learning in developing and deploying model capabilities.
- Teacher Knowledge Boundary: Explain why distillation cannot transfer a solution the teacher does not have, including the effect of retaining only correct teacher responses during rejection sampling.
Why it matters: This explains why making a model smaller and more practical does not, by itself, move the knowledge frontier. - Reasoning Style Versus New Knowledge: Distinguish improved accuracy from a genuinely new capability when distilling a teacher’s reasoning patterns into a base model.
Why it matters: A better reasoning style should not automatically be mistaken for an expanded set of capabilities. - Discovery And Deployment Roles: Describe the complementary roles assigned to reinforcement learning and distillation, and why neither replaces the other.
Why it matters: The two approaches address different needs: advancing what models can do and making those advances accessible.
Key terms: distillation, teacher, student, rejection sampling, correct reasoning traces, base model, reasoning patterns, accuracy, capability, new knowledge
8.8.3 The reproducibility gap
The section explains why DeepSeek’s distilled models can be used and broadly emulated, but their published results are difficult to reproduce exactly.
- Distillation reproducibility gap: Explain why releasing the distilled models and some training settings does not make DeepSeek’s results exactly reproducible, while still allowing others to train competitive models.
Why it matters: It helps distinguish access to a model from the ability to replicate the data and training process that produced it.
Key terms: reproducibility gap, SFT-based distillation, 800K training dataset, rejection sampling, quality filters, training hyperparameters
8.8.4 Looking ahead
This section explains several open directions for improving distillation and closes with the practical lesson that distillation brings frontier-model capabilities to smaller, deployable models.
- On-policy distillation trade-off: Distinguish static, off-policy teacher data from on-policy distillation, in which a student generates attempts that the teacher corrects or scores during training. Explain why this can target the student's own weaknesses but costs more.
Why it matters: The source of training examples determines whether feedback addresses the student's actual mistakes. - Further performance research questions: Describe RL after distillation and distillation scaling as open directions: DeepSeek did not apply RL to its distilled models, and a larger teacher may yield diminishing benefit for a small student.
Why it matters: Neither adding RL nor enlarging the teacher is presented as an automatic guarantee of better distillation. - Multiple and self teacher sources: Distinguish multi-teacher distillation, which combines teachers with different strengths, from self-distillation, which trains a model on selected best outputs of its own.
Why it matters: These approaches change where useful training examples come from, rather than simply making one teacher larger. - Distillation as deployment bridge: Explain the section's concluding lesson that distillation transfers knowledge from a massive model into much smaller models, connecting frontier research to practical deployment rather than treating scale alone as the measure of impact.
Why it matters: The chapter's payoff is not just a smaller model, but wider access to capabilities developed at the frontier.
Key terms: on-policy distillation, off-policy, static teacher data, RL after distillation, scaling laws for distillation, diminishing returns, multi-teacher distillation, self-distillation, rejection sampling, best-of-N
8.9 Summary
Concepts explained in this chapter
- distillationKnowledge distillation is a model-training technique in which a student model learns to reproduce information or behavior provided by a teacher model.
- dark knowledgeDark knowledge is the informative structure in a teacher model's output probabilities beyond the most likely prediction, including relationships among alternative outputs.
- student modelA distilled model is a student model trained to reproduce capabilities or behavior transferred from a larger teacher model.
- temperatureTemperature is a scalar parameter that controls the sharpness of a probability distribution produced from model logits.
- teacher modelA teacher model is a model whose outputs or learned behavior provide supervision for training another model.
- soft targetA soft target is a target distribution that assigns graded values or probabilities to multiple possible classes rather than selecting only one class.
- Kullback–Leibler divergenceKL divergence is a measure of the discrepancy between two probability distributions that quantifies information lost when one distribution approximates another.
- probability distributionA probability distribution assigns nonnegative probabilities to possible outcomes, with the probabilities summing to one.
- chain-of-thought promptingChain-of-thought is a sequence of intermediate natural-language reasoning steps produced between a problem statement and its final answer.
- DeepSeek-R1-Distill-Qwen-1.5BDeepSeek-R1-Distill-Qwen-1.5B is a 1.5-billion-parameter language model fine-tuned on reasoning data generated from DeepSeek-R1.
- scaling lawA scaling law is an empirical relationship describing how a model's performance or other properties change as resources such as model size increase.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent improves its behavior by optimizing rewards received from interactions or evaluations.
- datasetA dataset is a collection of data examples organized for analysis or for training, validating, or evaluating machine-learning models.
- expressive powerRepresentational capacity is the ability of a model architecture to represent functions, patterns, or input-output relationships.
- fine-tuningFine-tuning is continued training of a pretrained model on a targeted dataset to adapt its behavior or capabilities.
- foundation modelA base model is a pretrained model whose existing parameters and capabilities provide the foundation for subsequent adaptation or distillation.
- distillation lossA distillation loss is a training objective that measures or combines discrepancies between a student's predictions and information supplied by a teacher model.
- parameter countParameter count is the number of learned numerical parameters in a machine-learning model.
- hard-label lossHard-label loss is the cross-entropy error between a student's predicted token or class distribution and the ground-truth label.
- soft-target lossSoft-target loss is a divergence-based error between the probability distributions produced by a teacher and a student after temperature adjustment.
- total lossTotal loss is an aggregate training error formed by summing the individual losses associated with multiple prediction targets.
- DeepSeek distillation datasetThe DeepSeek distillation dataset is a book-specific collection of approximately 800,000 curated reasoning and non-reasoning training samples.
- rejection samplingRejection sampling is a data curation procedure that generates multiple candidates and retains only those meeting specified correctness and quality criteria.
- model compressionModel compression is the reduction of a machine-learning model's size or computational requirements while attempting to preserve its behavior.
- data explorationExploration is the process of trying varied actions or strategies to discover which ones produce desirable outcomes during learning.
- reasoning patternA reasoning pattern is a recurring strategy for carrying out multi-step inference or problem solving.
- DeepSeek-R1DeepSeek-R1 is a reasoning-focused large language model designed to achieve strong performance on demanding reasoning tasks.
- parameterA model parameter is a learned numerical value that determines part of a machine-learning model's computation.
- softmaxSoftmax is a function that converts a vector of scores into nonnegative values summing to one.
- pipelineA training pipeline is an organized sequence of data preparation, model computation, and optimization operations for training a machine-learning model.
- training loopA training loop is an iterative procedure that processes batches, computes predictions and loss, backpropagates gradients, updates parameters, and clears gradients.
- chain-of-thoughtReasoning behavior is a model's capability to carry out or express multi-step inference and problem-solving strategies.
- baselineA baseline model is a reference model trained under a simpler or comparison-only procedure without teacher guidance.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
