Distillation trains a smaller student model to approximate a larger teacher’s behavior, reducing inference cost while transferring useful outputs or capabilities. section 8.2.1
Choose the transferred signal according to the target behavior:
- Logit-based distillation matches the teacher’s probability distribution over possible next tokens rather than only hard labels. section 8.2.1
- Rationale distillation transfers explanations or reasoning traces alongside the task behavior. section 8.2.1
- Judge-filtered distillation uses teacher-generated evaluations or labels to train a cheaper target model. section 4.4.4 section 8.2.1
Distilling a cheaper evaluation judge
Evaluate a dataset with GPT-4, retain reasoning traces and scores, fine-tune a smaller model such as Qwen3 8B or Olmo3 7B, then validate it against a golden set before deployment. This can reduce inference cost by 10–100×, but requires estimating effort and monitoring agreement continuously. section 4.4.4
Figure 8.1 The Constitutional AI two-phase pipeline. Phase 1 generates critique-and-revise traces from a helpful-only baseline guided by the constitution, then fine-tunes the pretrained model on the <prompt, revised responses> pairs to produce the SL-CAI model. Phase 2 samples paired responses from the SL-CAI model, asks the model to pick the preferred response under the constitution, trains a reward model on those AI preferences, and runs PPO with a KL penalty against the SL-CAI reference. The downstream RL machinery is identical to InstructGPT.
Trace how critique-and-revise distillation produces an SL-CAI model before AI preferences train a reward model and PPO policy. section 8.2.1













