A teacher model is a larger or more capable model whose outputs supervise a smaller student model. Distillation transfers behavior without exposing the teacher’s internal logits. section 9.3.1
Teacher models can supervise students in several ways:
- Classical distillation matches the teacher’s output probability distributions. section 9.3.1
- Behavioral distillation fine-tunes the student with cross-entropy loss on labeled text generated by the teacher. section 9.3.1
- A critic model can grade retrieved passages and produce supervision for adaptive retrieval and self-critique. section 9.3.1
- GPT-4 is one option; open-weight models support privacy, on-premises deployment, and lower token costs, while domain-specific critics can better judge specialized corpora. section 9.3.1
Retrieval supervision
A critic evaluates retrieved passages, labels their groundedness, and generates examples that train a student to decide when to retrieve and when to critique its answer. section 9.3.1





