liveBook · Manning

Distillation

As explained in Build a DeepSeek Model (From Scratch)

Knowledge distillation is a model-training technique in which a student model learns to reproduce information or behavior provided by a teacher model.

Knowledge distillation trains a smaller student model to reproduce information or behavior from a stronger teacher. The goal is to make useful capabilities cheaper to run without training every small model independently from scratch. Compression transfers behavior; it does not require converting the teacher’s parameters into student parameters. section 8.2 section 8.1.2

Classical distillation uses a frozen teacher’s output distribution as a soft target. Unlike a hard label, the distribution gives the student information about alternatives as well as the preferred answer. Raising the softmax temperature spreads probability across those alternatives while preserving their ranking. section 8.2 section 8.4.3

A classical distillation training step combines two signals:

  • Compute the student’s hard-label loss to ground its predictions in the correct answers. section 8.4.3
  • Compare teacher and student distributions at temperature T using KL divergence. Combine the losses as L = αL_hard + βT²L_soft; the T² factor corrects for temperature scaling. section 8.4.3 section 8.5.3

For reasoning models, distillation can instead transfer complete generated responses. A teacher produces reasoning traces offline; rejection sampling curates them, and a pretrained student learns from the resulting prompt–response pairs using ordinary next-token supervised fine-tuning. This route does not use the teacher’s logits, KL divergence, or temperature scaling during student training. section 8.5.3 section 8.1.2

Figure 8.19 Two paradigms of knowledge distillation. Classical KD (left) transfers knowledge through logit-matching with KL divergence and temperature scaling. DeepSeek’s approach (right) transfers knowledge through supervised fine-tuning on the teacher’s generated chain-of-thought text.

Compare simultaneous soft-target matching with offline training on generated reasoning traces. section 8.5.3

Listing 8.3 Rejection sampling for reasoning data curation.
def rejection_sample(teacher, prompts,
     [CA] n_samples=4):  #A
    dataset = []
    for prompt in prompts:
        candidates = []
        for _ in range(n_samples):  #B
            response = teacher.generate(
             [CA] prompt, temperature=0.6,
             [CA] top_p=0.95,
             [CA] max_tokens=32768)
            candidates.append(response)
        for resp in candidates:
            answer = extract_answer(resp)
            truth = get_ground_truth(
             [CA] prompt)  #C
            if verify(answer, truth):
                if is_clean(resp):  #D
                    dataset.append(
                     [CA] (prompt, resp))
                    break
    return dataset  #E

Trace a rejection-sampling loop that generates candidates, verifies answers, and retains clean responses. section 8.5.5

DeepSeek-R1 reasoning transfer

DeepSeek-R1 generated a curated set of 800,000 reasoning examples used to fine-tune six Qwen- and Llama-based students. On the same Qwen2.5-32B base, the distilled model scored 72.6% on AIME 2024 versus 47.0% for a direct RL-trained model. This comparison does not make distillation universally superior to RL: RL can strengthen a teacher, while distillation distributes its existing behavior. section 8.1.2 section 8.7.2 section 7.7.6

Distillation is not lossless. The 1.5B student scored 83.9% on MATH-500 but 16.9% on LiveCodeBench, and the teacher remained ahead of the distilled models in the reported comparisons. Student capacity, base model, and task all matter; distillation generally transfers behavior the teacher already has rather than discovering new behavior beyond it. section 8.7.1 section 7.7.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Develops temperature-scaled soft-target loss and a runnable training loop, then contrasts them with offline reasoning-trace fine-tuning.
Examples
Compares hard-label and distilled toy students and reports DeepSeek-R1’s Qwen and Llama reasoning benchmarks.
What it assumes
Introduces concepts with minimal prior assumptions and develops distillation from first principles using traceable toy models.
What you can do after
Build a classical distillation pipeline from scratch in PyTorch and evaluate the trade-offs of reasoning-trace distillation.
How it connects
Connects distillation to supervised fine-tuning, reinforcement learning, rejection sampling, temperature scaling, and KL divergence.

Also explained in

Compare how 14 books explain distillation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.