liveBook · Manning

Distillation

As explained in LLM Customization and Fine-Tuning

Training a student model to imitate the behavior of a teacher model, often a larger or stronger one.

A strong API model may answer support questions well but become expensive at scale or require sending sensitive requests off-network. Distillation trains a student model to reproduce a teacher’s behavior, often in a smaller, locally served model that does not need the teacher at inference time. The tradeoff is an upfront investment in data generation and training, with no guarantee that the student will retain enough quality. section 7.0 section 7.3

In black-box distillation, the teacher generates responses to task prompts and the student is trained on the resulting pairs using supervised fine-tuning. The distinguishing feature is the source of supervision, not a new training algorithm. Teacher-generated examples can also extend coverage to rare scenarios and edge cases, but a consistent teacher can pass along consistent errors. section 7.4.4 section 7.4.2 section 7.4.3

A practical pipeline has four decisions:

  • Generate representative prompts and teacher responses, including difficult cases. Check the teacher’s terms before using API outputs for model training, and control which prompts leave your infrastructure. section 7.4.2 section 7.6.7 section 7.3
  • Filter terse, repetitive, or otherwise low-quality outputs and verify their correctness. The student can reproduce the teacher’s mistakes as well as its useful patterns. section 7.9 section 7.4.3
  • Train the student, often with a LoRA adapter. If adapter capacity is insufficient for the task, consider full supervised fine-tuning instead. section 7.1.4
  • Compare base, teacher, and student on the same held-out tasks. Inspect category-level performance and behavior, not just an aggregate overlap score, and test the student independently for safety regressions. section 7.0 section 7.2.2 section 7.9

Black-box distillation is the practical default when the teacher is a proprietary API: only its text responses are needed. White-box distillation uses the teacher’s token distribution and requires access to its weights. Consider it when both models are under your control and the richer signal is worth the infrastructure, particularly for demanding capability or reasoning targets. section 7.4.4

IT support student

On 50 held-out support questions, a LoRA student reproduced the teacher’s Summary-plus-numbered-Steps format in all 50 answers; the unadapted base did so in none. The student reached about 98% of the teacher’s Token-F1, but that overlap metric does not establish safety or generalize retention to every task. In the same run, the student lost harmful-request refusals the base model had. section 7.2.2 section 7.9

Table 7.1 Three-way distillation evaluation results
Model
Overall Token-F1
Relative to the Teacher
Base (Qwen3-4B, no adaptation)
0.153
91%
Teacher (Ch6 SFT model)
0.168
100% (reference)
Student (LoRA on teacher data)
0.165
98%

Compare the base, teacher, and student scores from the held-out support evaluation. section 7.2.2

Choose distillation when a stable, sufficiently frequent task justifies upfront work and the measured quality gap is acceptable. Retention depends on task difficulty and student capacity: hard reasoning may require a larger student or a different technique. If the only requirement is a fixed output shape, a decode-time schema can enforce it without training. For changing knowledge, consider retrieval instead. section 7.6.7 section 7.4.5 section 7.2.2 section 1.6.2

For production lineage, record the teacher identifier and version, prompt set, filtering parameters, student training settings, and evaluation results. A locally served student avoids sending each inference request to the teacher, but any teacher API calls made during data generation still require their own privacy review. section 7.6.7 section 7.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Walks through teacher-data generation, filtering, student training, and three-way evaluation, with black-box and white-box tradeoffs.
Examples
Uses an IT support assistant and DeepSeek-R1 student benchmarks to illustrate behavioral and task-dependent retention.
What it assumes
Reuses supervised fine-tuning, LoRA, and chat-template workflows while introducing teacher-generated supervision.
What you can do after
Builds a black-box student pipeline and evaluates its retention and safety against base and teacher models.
How it connects
Connects distillation to supervised fine-tuning, LoRA, retrieval-augmented generation, and safety alignment.

Also explained in

Compare how 14 books explain distillation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.