Distillation trains a student model on supervision produced by a stronger teacher. In language-model post-training, that supervision can be completions, labels, logits, critiques, or preference signals. The practical goal is usually to transfer useful behavior or a targeted skill into a smaller model, while using synthetic data at a scale unavailable from human-written answers alone. section 12.2 section 12.3
Two related meanings matter. Traditional knowledge distillation makes the student match the teacher’s soft probability distribution rather than a one-hot target. Colloquial LLM distillation commonly means collecting teacher-generated outputs and fine-tuning a student on them with standard supervised learning. Both use a strong model as the source of training signal, but the first directly matches distributions while the second can reduce to learning generated sequences. section 12.2 section 12.3.1
Figure 12.1 Traditional knowledge distillation trains a smaller student model to match the soft probability distribution of a larger teacher model using KL divergence loss. Both models process the same input simultaneously, and temperature scaling (\(\tau > 1\)) softens the distributions to reveal more information about class relationships.
Soft-label distillation runs teacher and student on the same input, softens their distributions with temperature, and minimizes KL-based distribution mismatch. The softened probabilities expose relationships among alternatives that a single correct label omits. section 12.2
For an autoregressive language model, per-token distillation compares teacher and student next-token distributions at every output position. Given prompt s and teacher sequence u, the teacher assigns q(k | s, u<j) across vocabulary tokens and the student assigns p(k | s, u<j). Cross-entropy penalizes the student when it assigns low probability to tokens the teacher considers likely; minimizing it is equivalent to minimizing forward KL divergence up to the teacher distribution’s fixed entropy. section 12.3.1
Equation 12.1
\[\label{eq:word_kd} \mathcal{L}_{\mathrm{WORD-KD}} = -\sum_{j=1}^{J}\sum_{k=1}^{|\mathcal{V}|} q(u_j = k \mid s, u_{<j})\log p(u_j = k \mid s, u_{<j}).\]The word-level objective sums teacher-weighted negative log probabilities across each output position and every vocabulary token. For modern models, “word-level” means per-token matching over the tokenizer vocabulary. section 12.3.1
Sequence-level distillation instead targets complete outputs. Exact matching over all possible sequences is impractical, so it can be approximated by training on a teacher-generated sequence. This approximation aligns naturally with a synthetic-data pipeline: pass curated prompts to a teacher, pair prompts with completions, then fine-tune the student. More elaborate pipelines can edit completions, produce preference pairs, or filter output quality before training. section 12.2 section 12.3.1
Distillation is used in post-training in two broad ways:
- Run a data engine across instruction tuning, preference or constitutional data, verification for reinforcement learning, and related supervision pipelines. section 12.2
- Transfer a particular capability, notably mathematical reasoning or coding, from a stronger model into a weaker one. section 12.2
- Train a reasoning model after another model has received substantial RLVR training, using that model’s outputs as large-scale instruction-tuning data. section 7.3
Data quality remains the central implementation constraint. Curating prompts and filtering teacher responses are crucial for performance. Synthetic completions have largely displaced human-written completions for instruction tuning where models are more reliable, but human data remains important for frontier capabilities and ground-truth answers. Synthetic preference feedback is cheaper and easier to scale, yet its low-noise, high-bias character can introduce systematic effects. section 12.2 section 12.7 section 12.1
Teacher completion to student fine-tuning
A laboratory can keep a large internal model unreleased, ask it to answer a curated set of prompts, filter the resulting completions, and fine-tune a smaller model on the prompt–completion pairs. Open-model workflows similarly distill data from closed API models into smaller publicly available weights. section 12.2
