A teacher model is a powerful, resource-intensive model whose outputs guide a smaller student model. In sequence-level distillation, the teacher answers domain-specific prompts; the student is fine-tuned on those input–output pairs to imitate its behavior. Unlike classic distillation, this approach does not require access to the teacher’s probability outputs or logits. section 5.4 section 5.4.1
To use a teacher for production distillation:
- Choose a strong teacher and collect varied, high-quality responses to prompts that reflect real use. section 5.4.2
- Fine-tune the smaller student on the resulting prompt–response pairs, then evaluate tone, format, and correctness. section 5.4.1 section 5.4.2
- Regenerate training data and retrain as the domain or user needs change. section 5.4.2
Customer-support distillation
A flagship model answers e-commerce support questions, such as how to return a damaged product. Those answers become training examples for a smaller student when serving the flagship at scale is impractical. section 5.4 section 5.4.1
{
"messages": [
{ "role": "user", "content": "How do I return a damaged product?" },
{ "role": "assistant", "content": "Sorry to hear that! You can return any
damaged item within 30 days by visiting our Returns Center..." }
]
}
Example user–assistant exchange for a damaged-product return. section 5.4.1
