liveBook · Manning

Teacher model

As explained in LLM Customization and Fine-Tuning

A model whose outputs or behavior provide targets for training a student model during distillation.

A teacher model produces outputs that serve as training targets for a student model. In black-box distillation, the student learns from the teacher’s text responses through supervised fine-tuning, then runs without consulting the teacher at inference time. The teacher can be a frontier API, a larger open-weights model, or an already fine-tuned model. section 7.0 section 7.9

The decision is whether the student preserves enough task-specific behavior to justify a cheaper deployment. Distillation can transfer response structure, reasoning patterns, uncertainty and level of detail, but it can also copy the teacher’s mistakes. A locally run student may reduce inference cost and avoid sending each production request to an external API. section 7.4.1 section 7.3.1 section 7.0

Use teacher outputs as training data deliberately:

  • Anchor synthetic examples to two or three high-quality real examples so they match the task’s style, vocabulary and structure. section 3.4.3
  • Filter degenerate outputs, including terse or repetitive responses. Judge fabrications against the task: an invented hostname may be harmless for ticket classification but dangerous in a summary that presents it as fact. section 7.9 section 3.4.3
  • When augmenting real data with synthetic examples, use 30% as a first-run cap rather than making synthetic data the primary source. section 3.4.3
  • Compare base, teacher and student on the same held-out data. Check behavior and safety as well as lexical metrics; test each student independently for safety regressions. section 7.2.2 section 7.9

IT support assistant

On 50 held-out examples, a fine-tuned teacher scored 0.168 Token-F1, compared with 0.153 for the base model and 0.165 for the student. The student used a 130 MB LoRA adapter rather than the teacher’s full 8 GB model. Its score was about 98% of the teacher’s, but Token-F1 measures word overlap, not whether behavior or safety transferred. In this run, the student matched the house format but lost harmful-request refusals the base model had. section 7.2.2 section 7.9

Table 7.1 Three-way distillation evaluation results
Model
Overall Token-F1
Relative to the Teacher
Base (Qwen3-4B, no adaptation)
0.153
91%
Teacher (Ch6 SFT model)
0.168
100% (reference)
Student (LoRA on teacher data)
0.165
98%

Compare the base, teacher and student scores on the same held-out test split. section 7.2.2

Multiple teachers can provide a way to draw on different strengths rather than rely on one teacher’s blind spots. For black-box teachers, their outputs can be pooled before training; reconciling tokenizers across white-box model families remains a research problem. section 7.3.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains teacher selection, output generation and filtering, followed by three-way evaluation and safety checks.
Examples
Uses an IT support assistant and a 50-example held-out comparison of base, teacher and student models.
What it assumes
Builds on familiarity with supervised fine-tuning while defining distillation as training on a stronger model’s outputs.
What you can do after
Evaluate a teacher-student setup using behavioral, lexical and safety checks rather than assuming the student inherits every teacher capability.
How it connects
Connects knowledge distillation with supervised fine-tuning, synthetic data generation and safety alignment.

Also explained in

Compare how 6 books explain teacher model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.