A strong API model may answer support questions well but become expensive at scale or require sending sensitive requests off-network. Distillation trains a student model to reproduce a teacher’s behavior, often in a smaller, locally served model that does not need the teacher at inference time. The tradeoff is an upfront investment in data generation and training, with no guarantee that the student will retain enough quality. section 7.0 section 7.3
In black-box distillation, the teacher generates responses to task prompts and the student is trained on the resulting pairs using supervised fine-tuning. The distinguishing feature is the source of supervision, not a new training algorithm. Teacher-generated examples can also extend coverage to rare scenarios and edge cases, but a consistent teacher can pass along consistent errors. section 7.4.4 section 7.4.2 section 7.4.3
A practical pipeline has four decisions:
- Generate representative prompts and teacher responses, including difficult cases. Check the teacher’s terms before using API outputs for model training, and control which prompts leave your infrastructure. section 7.4.2 section 7.6.7 section 7.3
- Filter terse, repetitive, or otherwise low-quality outputs and verify their correctness. The student can reproduce the teacher’s mistakes as well as its useful patterns. section 7.9 section 7.4.3
- Train the student, often with a LoRA adapter. If adapter capacity is insufficient for the task, consider full supervised fine-tuning instead. section 7.1.4
- Compare base, teacher, and student on the same held-out tasks. Inspect category-level performance and behavior, not just an aggregate overlap score, and test the student independently for safety regressions. section 7.0 section 7.2.2 section 7.9
Black-box distillation is the practical default when the teacher is a proprietary API: only its text responses are needed. White-box distillation uses the teacher’s token distribution and requires access to its weights. Consider it when both models are under your control and the richer signal is worth the infrastructure, particularly for demanding capability or reasoning targets. section 7.4.4
IT support student
On 50 held-out support questions, a LoRA student reproduced the teacher’s Summary-plus-numbered-Steps format in all 50 answers; the unadapted base did so in none. The student reached about 98% of the teacher’s Token-F1, but that overlap metric does not establish safety or generalize retention to every task. In the same run, the student lost harmful-request refusals the base model had. section 7.2.2 section 7.9
Table 7.1 Three-way distillation evaluation results
|
Model
|
Overall Token-F1
|
Relative to the Teacher
|
|
Base (Qwen3-4B, no adaptation)
|
0.153
|
91%
|
|
Teacher (Ch6 SFT model)
|
0.168
|
100% (reference)
|
|
Student (LoRA on teacher data)
|
0.165
|
98%
|
Compare the base, teacher, and student scores from the held-out support evaluation. section 7.2.2
Choose distillation when a stable, sufficiently frequent task justifies upfront work and the measured quality gap is acceptable. Retention depends on task difficulty and student capacity: hard reasoning may require a larger student or a different technique. If the only requirement is a fixed output shape, a decode-time schema can enforce it without training. For changing knowledge, consider retrieval instead. section 7.6.7 section 7.4.5 section 7.2.2 section 1.6.2
For production lineage, record the teacher identifier and version, prompt set, filtering parameters, student training settings, and evaluation results. A locally served student avoids sending each inference request to the teacher, but any teacher API calls made during data generation still require their own privacy review. section 7.6.7 section 7.3













