liveBook · Manning

Teacher model

As explained in Retrieval Augmented Generation, The Foundational Ideas

A teacher model is a larger or more capable model whose outputs or behavior provide supervision for training a student model.

A teacher model is a larger or more capable model whose outputs supervise a smaller student model. Distillation transfers behavior without exposing the teacher’s internal logits. section 9.3.1

Teacher models can supervise students in several ways:

  • Classical distillation matches the teacher’s output probability distributions. section 9.3.1
  • Behavioral distillation fine-tunes the student with cross-entropy loss on labeled text generated by the teacher. section 9.3.1
  • A critic model can grade retrieved passages and produce supervision for adaptive retrieval and self-critique. section 9.3.1
  • GPT-4 is one option; open-weight models support privacy, on-premises deployment, and lower token costs, while domain-specific critics can better judge specialized corpora. section 9.3.1

Retrieval supervision

A critic evaluates retrieved passages, labels their groundedness, and generates examples that train a student to decide when to retrieve and when to critique its answer. section 9.3.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains classical and behavioral distillation, then compares critic choices and deployment tradeoffs.
Examples
Uses GPT-4, Llama 3 70B, Qwen 2.5 72B, Mixtral 8x22B, and domain-specific critics as alternatives.
What it assumes
Assumes familiarity with fine-tuning and cross-entropy loss while defining distillation from first principles.
What you can do after
Evaluate/understand how teacher-generated examples transfer retrieval and critique behavior to a deployable student.
How it connects
Connects teacher models to student models, knowledge distillation, retrieval, reflection tokens, and self-critique.

Also explained in

Compare how 6 books explain teacher model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.