liveBook · Manning

Chapter 8: Knowledge distillation: Making powerful models practical

What you will learn in chapter 8 of Build a DeepSeek Model (From Scratch)

The introduction motivates knowledge distillation as a way to make DeepSeek-R1’s reasoning ability practical to deploy in much smaller models.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction motivates knowledge distillation as a way to make DeepSeek-R1’s reasoning ability practical to deploy in much smaller models.

Key terms: DeepSeek-R1, knowledge distillation, distilled model

8.1 Why 671 billion parameters won’t fit in your pocket

The section frames deployment as a challenge distinct from building a powerful language model.

Key terms: deploying

8.1.1 The cost of intelligence

This section quantifies why the full DeepSeek-R1 model is costly to run and explains the gap between its reasoning capability and practical deployment.

Key terms: Mixture-of-Experts, active parameters, model weights, FP16, inference, KV Cache, deployment gap, cloud API, quantization, pruning

8.1.2 The dream: Reasoning in your pocket

This section shows that a small distilled model can make some of DeepSeek-R1’s reasoning capabilities practical to run, while clarifying what is transferred and where the smaller model falls short.

Key terms: 671B model, DeepSeek-R1-Distill-Qwen-1.5B, AIME 2024, coding tasks, distilled models, Qwen2.5, Llama-3, 800K curated reasoning examples, fine-tune

8.2 The teacher-student paradigm

This section introduces knowledge distillation as training a small student model to reproduce a larger teacher model’s behavior using the teacher’s output probabilities.

Key terms: knowledge distillation, teacher, student, model compression, frozen teacher, soft targets, hard labels, full probability distribution

8.2.1 Why soft labels carry more information than hard labels

Explains why a teacher’s soft probability distribution gives a student more useful information than a hard label.

Key terms: hard labels, one-hot vector, soft labels, dark knowledge, inter-class similarity relationships

8.2.2 The master chef analogy

This section uses a cooking analogy to explain why a teacher’s soft targets can help a student generalize beyond what hard labels teach, then illustrates distillation’s practical value and its earlier scope.

Key terms: hard label, soft targets, teacher’s probability distribution, generalization, DistilBERT, soft-target distillation, intermediate layer matching, hidden states, fixed set of output classes, reasoning

8.3 Temperature and dark knowledge

This section explains why a peaked teacher output can hide useful relationships between incorrect classes and introduces temperature scaling as a way to reveal them.

Key terms: incorrect classes, dark knowledge, logits, temperature scaling, softmax, T=1

8.3.1 The temperature-scaled softmax

This subsection explains how temperature changes softmax probabilities and why moderate softening can reveal useful information for distillation.

Key terms: temperature, temperature-scaled softmax, softness, inter-class similarity structure

8.3.2 A numerical walkthrough: seeing the dark knowledge emerge

A four-class example shows how temperature changes a teacher’s softmax probabilities and why a moderately softened distribution reveals useful class relationships.

Key terms: logits, temperature, softmax, dark knowledge, entropy, useful entropy, Goldilocks zone

8.3.3 What the dark knowledge reveals

This section explains how temperature-softened teacher probabilities reveal relationships between classes and give a student richer supervision than hard labels.

Key terms: dark knowledge, soft targets, hard labels, inter-class similarity structure

8.3.4 Why T² scaling matters

Explains why temperature-scaled distillation multiplies the soft-target loss by T² to keep its learning signal strong.

Key terms: T² scaling factor, KL loss, soft-target loss, temperature, dark knowledge

8.3.5 The temperature Goldilocks zone

Explains how to choose a distillation temperature that reveals useful dark knowledge without overwhelming the student model.

Key terms: temperature, Goldilocks zone, dark knowledge, distribution, student, capacity

8.4 Building the distillation loss: From naive to powerful

Introduces a progression of three approaches that will build toward the complete knowledge distillation loss.

8.4.1 Attempt #1: Training from scratch with hard labels

The section explains why training a student from scratch on hard labels is a slow baseline compared with learning from a teacher’s soft targets.

Key terms: student model, hard labels, one-hot label, soft targets, inter-class relationships

8.4.2 Attempt #2: Matching the teacher’s output at T=1

This section explains why training a student on the teacher’s output probabilities at T=1 yields only a small improvement over hard labels.

Key terms: teacher’s output probabilities, T=1, hard label, gradients, inter-class structure, temperature

8.4.3 Temperature-scaled soft targets

This section shows how raising softmax temperature makes the teacher’s soft targets more informative and explains why distillation also needs hard labels.

Key terms: temperature, soft targets, dark knowledge, inter-class relationships, hard labels, grounding

8.4.4 The complete distillation loss

This section combines hard-label and teacher-guided losses into a classical distillation objective and explains how it trains only the student.

Key terms: hard-label loss, soft-target loss, cross-entropy, KL divergence, dark knowledge, T² scaling, α, β, gradient magnitude, teacher model

8.5 DeepSeek-R1’s distillation recipe

This section contrasts fixed-class knowledge distillation with language generation to frame the challenge of distilling reasoning.

Key terms: classical knowledge distillation, output space, probability distribution, token by token, chains of reasoning

8.5.1 From logits to language: a paradigm shift

This section explains why DeepSeek-R1 distillation transfers generated reasoning traces rather than matching the teacher’s token probabilities.

Key terms: chain-of-thought (CoT) reasoning traces, per-token probability distribution, coherent reasoning strategy, chain-of-thought distillation, supervised fine-tuning, next-token prediction, logit distribution, KL divergence, temperature scaling

8.5.2 The 800K training dataset: Rejection sampling at scale

This section explains how DeepSeek curated roughly 800K teacher-generated examples for distillation, balancing verified reasoning traces with general-purpose responses.

Key terms: rejection sampling, candidate responses, distillation dataset, ground truth, DeepSeek-V3, generative reward model, reasoning samples, non-reasoning samples, illustrative estimate, reasoning traces

8.5.3 Classical KD versus DeepSeek’s approach

This section contrasts classical logit-matching distillation with DeepSeek’s offline, text-based approach and explains why the latter simplifies student training.

Key terms: classical KD, logits, KL divergence, temperature scaling, T² correction, rejection sampling, soft targets, offline, static dataset, SFT

8.5.4 The six distilled models

This section identifies DeepSeek-R1’s six distilled models and explains how base-model choice and fine-tuning settings affect the resulting students.

Key terms: Qwen2.5, Llama 3, Qwen2.5-Math, math-specialized base, general-purpose bases, 800K dataset, cosine learning rate decay, maximum context length, learning rates

8.5.5 The economics of distillation

This section explains why generating a reusable reasoning dataset from a costly teacher can be cheaper than training each small model independently with reinforcement learning.

Key terms: rejection sampling, distillation data, student models, standard SFT

8.6 Implementing knowledge distillation in PyTorch

Introduces a planned PyTorch knowledge distillation pipeline using small, traceable teacher and student models.

8.6.1 Defining teacher and student models

Defines toy teacher and student models that share an input and output setup but differ substantially in capacity.

Key terms: teacher, student, embedding layer, output space, representational capacity

8.6.2 The training loop

This section walks through a distillation training step in which a frozen teacher and trainable student process the same inputs, and a combined loss updates only the student.

Key terms: frozen teacher model, student model, torch.no_grad(), backpropagation, hard-label loss, soft-target loss, cross-entropy, KL divergence, temperature, T²

8.6.3 Comparing the three approaches

Compares three ways to evaluate a student model and explains where distillation provides the greatest benefit.

Key terms: baseline, hard labels, soft targets, T=3, teacher (upper bound), dark knowledge, ambiguous, borderline examples, AIME

8.7 The payoff: An empirical head-to-head comparison

Introduces the upcoming empirical comparison of DeepSeek-R1’s distilled models without presenting the results.

8.7.1 The headline: 1.5B beats GPT-4o

This section compares DeepSeek-R1’s distilled models with frontier models on reasoning benchmarks, highlighting strong math results alongside limits in coding and teacher-level performance.

Key terms: DeepSeek-R1-Distill-Qwen-1.5B, AIME 2024, GPT-4o, Claude-3.5-Sonnet, Qwen distilled model, o1-mini, MATH-500, AIME, GPQA Diamond, LiveCodeBench

8.7.2 The critical comparison: Distillation versus direct RL

This section compares distillation with direct reinforcement learning on the same 32B base model and explains what the results imply for building capable smaller models.

Key terms: Qwen2.5-32B, DS-R1-Zero-32B, DS-R1-Distill-32B, SFT, AIME 2024, QwQ-32B, capacity, RL-from-scratch, distillation, large-scale reinforcement learning

8.7.3 Scaling behavior

This section shows how distilled reasoning performance changes with student size and base model, and why distillation succeeds where direct reinforcement learning with small models struggled.

Key terms: smooth scaling, model size, AIME 2024, base model, pre-existing knowledge, Qwen2.5-Math, Llama, direct RL, exploration, imitation

8.8 The limits of distillation

8.8.1 The capacity gap

This section explains why a small distilled student can retain the teacher’s abilities unevenly across tasks and uses that capacity gap to guide student-model size choices.

Key terms: capacity gap, student model, teacher, MATH-500, LiveCodeBench, 1.5B–3B, 7B–14B, 32B–70B, domain-specific, general-purpose reasoning

8.8.2 The knowledge frontier

This section explains why distillation is bounded by the teacher’s knowledge and how it complements reinforcement learning in developing and deploying model capabilities.

Key terms: distillation, teacher, student, rejection sampling, correct reasoning traces, base model, reasoning patterns, accuracy, capability, new knowledge

8.8.3 The reproducibility gap

The section explains why DeepSeek’s distilled models can be used and broadly emulated, but their published results are difficult to reproduce exactly.

Key terms: reproducibility gap, SFT-based distillation, 800K training dataset, rejection sampling, quality filters, training hyperparameters

8.8.4 Looking ahead

This section explains several open directions for improving distillation and closes with the practical lesson that distillation brings frontier-model capabilities to smaller, deployable models.

Key terms: on-policy distillation, off-policy, static teacher data, RL after distillation, scaling laws for distillation, diminishing returns, multi-teacher distillation, self-distillation, rejection sampling, best-of-N

8.9 Summary

Concepts explained in this chapter

← Chapter 7: Reinforcement learning: From policy gradients to GRPO

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.