A teacher model produces outputs that serve as training targets for a student model. In black-box distillation, the student learns from the teacher’s text responses through supervised fine-tuning, then runs without consulting the teacher at inference time. The teacher can be a frontier API, a larger open-weights model, or an already fine-tuned model. section 7.0 section 7.9
The decision is whether the student preserves enough task-specific behavior to justify a cheaper deployment. Distillation can transfer response structure, reasoning patterns, uncertainty and level of detail, but it can also copy the teacher’s mistakes. A locally run student may reduce inference cost and avoid sending each production request to an external API. section 7.4.1 section 7.3.1 section 7.0
Use teacher outputs as training data deliberately:
- Anchor synthetic examples to two or three high-quality real examples so they match the task’s style, vocabulary and structure. section 3.4.3
- Filter degenerate outputs, including terse or repetitive responses. Judge fabrications against the task: an invented hostname may be harmless for ticket classification but dangerous in a summary that presents it as fact. section 7.9 section 3.4.3
- When augmenting real data with synthetic examples, use 30% as a first-run cap rather than making synthetic data the primary source. section 3.4.3
- Compare base, teacher and student on the same held-out data. Check behavior and safety as well as lexical metrics; test each student independently for safety regressions. section 7.2.2 section 7.9
IT support assistant
On 50 held-out examples, a fine-tuned teacher scored 0.168 Token-F1, compared with 0.153 for the base model and 0.165 for the student. The student used a 130 MB LoRA adapter rather than the teacher’s full 8 GB model. Its score was about 98% of the teacher’s, but Token-F1 measures word overlap, not whether behavior or safety transferred. In this run, the student matched the house format but lost harmful-request refusals the base model had. section 7.2.2 section 7.9
Table 7.1 Three-way distillation evaluation results
|
Model
|
Overall Token-F1
|
Relative to the Teacher
|
|
Base (Qwen3-4B, no adaptation)
|
0.153
|
91%
|
|
Teacher (Ch6 SFT model)
|
0.168
|
100% (reference)
|
|
Student (LoRA on teacher data)
|
0.165
|
98%
|
Compare the base, teacher and student scores on the same held-out test split. section 7.2.2
Multiple teachers can provide a way to draw on different strengths rather than rely on one teacher’s blind spots. For black-box teachers, their outputs can be pooled before training; reconciling tokenizers across white-box model families remains a research problem. section 7.3.1





