Distillation adapts a smaller student LLM by training it on outputs from a larger teacher. For reasoning tasks, those outputs usually contain both intermediate reasoning traces and final answers. It trades the teacher’s deployment cost for a one-time dataset-generation and student-training workflow. section 8.1 section 8.2
Hard distillation treats teacher-generated tokens as supervised targets and usually uses cross-entropy loss. Soft distillation instead matches the teacher’s vocabulary probability distribution by minimizing KL divergence; it requires teacher logits, which are often unavailable and more expensive to store. section 8.1 section 8.11
A practical hard-distillation workflow proceeds as follows:
- Select problems that are separate from the evaluation set, then submit them to a stronger teacher and save its reasoning traces and answers. section 8.2
- Render each prompt with the teacher response, optionally separating reasoning with <think>...</think> tags; tokenize once and filter examples by maximum sequence length. section 8.11
- Load the pretrained student, shuffle examples each epoch, compute answer-only cross-entropy, backpropagate, update weights, clip large gradients when needed, and record validation loss. section 8.5 section 8.7
- Evaluate saved checkpoints on a held-out benchmark rather than relying only on training loss; validation loss is the more informative progress signal. section 8.8
DeepSeek-R1 to Qwen3 0.6B
A practical setup uses DeepSeek-R1 as the teacher and Qwen3 0.6B as the student. Teacher responses for 12,000 non-overlapping MATH training problems form the supervision, while MATH-500 remains a separate evaluation set. A full run takes about three hours and uses roughly 15 GB of GPU memory on a DGX Spark; shorter sequences reduce memory requirements. section 8.1 section 8.2 section 8.7
Figure 8.2 Hard distillation trains the student on teacher-generated tokens; soft distillation trains the student on the teacher’s full output distribution.
Use the comparison figure to distinguish token-target training in hard distillation from full-distribution matching in soft distillation. section 8.1
Figure 8.3 This chapter covers distillation in four main steps: (1) distillation introduction and overview; (2) dataset preparation (steps 2a-2c); (3) training via distillation (steps 3a-3c); and (4) evaluation.
Use the workflow figure to navigate the four stages: introduction, dataset preparation, distillation training, and evaluation. section 8.1
