liveBook · Manning

Distillation

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Distillation is a model-compression technique in which a smaller model is trained to reproduce the behavior or outputs of a larger model.

Distillation trains a smaller student model to approximate a larger teacher’s behavior, reducing inference cost while transferring useful outputs or capabilities. section 8.2.1

Choose the transferred signal according to the target behavior:

  • Logit-based distillation matches the teacher’s probability distribution over possible next tokens rather than only hard labels. section 8.2.1
  • Rationale distillation transfers explanations or reasoning traces alongside the task behavior. section 8.2.1
  • Judge-filtered distillation uses teacher-generated evaluations or labels to train a cheaper target model. section 4.4.4 section 8.2.1

Distilling a cheaper evaluation judge

Evaluate a dataset with GPT-4, retain reasoning traces and scores, fine-tune a smaller model such as Qwen3 8B or Olmo3 7B, then validate it against a golden set before deployment. This can reduce inference cost by 10–100×, but requires estimating effort and monitoring agreement continuously. section 4.4.4

Figure 8.1 The Constitutional AI two-phase pipeline. Phase 1 generates critique-and-revise traces from a helpful-only baseline guided by the constitution, then fine-tunes the pretrained model on the <prompt, revised responses> pairs to produce the SL-CAI model. Phase 2 samples paired responses from the SL-CAI model, asks the model to pick the preferred response under the constitution, trains a reward model on those AI preferences, and runs PPO with a KL penalty against the SL-CAI reference. The downstream RL machinery is identical to InstructGPT.

Trace how critique-and-revise distillation produces an SL-CAI model before AI preferences train a reward model and PPO policy. section 8.2.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains teacher-student transfer, three signal types, validation, and a two-phase pipeline without fully detailing every training objective.
Examples
Uses GPT-4-to-small-judge transfer and critique-and-revise Constitutional AI traces as concrete examples.
What it assumes
Assumes basic supervised learning and pretrained models, while introducing teacher-student transfer and alternative supervision signals.
What you can do after
Apply it by collecting teacher outputs, training a smaller model, validating against a golden set, and monitoring agreement after deployment.
How it connects
Connects model compression with supervised learning, reward models, preference learning, PPO, KL penalties, and Constitutional AI.

Also explained in

Compare how 14 books explain distillation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.