A capable support model may answer customer questions well yet be too slow or expensive to serve at scale. Knowledge distillation transfers its behavior to a smaller student model. The aim is to keep useful response quality while reducing inference cost, latency, and hardware demands. section 5.4 section 5.4.1
For LLMs, a practical approach is sequence-level distillation: send domain-specific prompts to the teacher, save its final responses, and fine-tune the student on those input–output pairs. Unlike classic distillation, this does not require access to the teacher’s probability outputs or logits. The student learns from what the teacher says, not from its internal calculations. section 5.4.1
{
"messages": [
{ "role": "user", "content": "How do I return a damaged product?" },
{ "role": "assistant", "content": "Sorry to hear that! You can return any
damaged item within 30 days by visiting our Returns Center..." }
]
}
A user question and a teacher-written answer form one training example; many varied, high-quality pairs are needed to represent real support traffic. section 5.4.1 section 5.4.2
A production workflow has four parts:
- Choose a strong teacher and collect domain-specific prompts that reflect real use. Review its answers: the student can learn only behavior the teacher demonstrates. section 5.4.1 section 5.4.2
- Fine-tune a smaller student on the captured prompt–response pairs. An OpenAI fine-tuning job can use a completions file to train a smaller model. section 5.4.1
- Evaluate the student against a support test set. Track accuracy and similarity, but also check tone, format, and correctness; resemblance to the teacher alone does not establish a correct answer. section 5.4.1 section 5.4.2
- Regenerate examples and retrain when the domain or user needs change. section 5.4.2
A smaller student can make deployment feasible on less demanding hardware, including smaller GPU instances, CPU-only servers, or edge devices. Distillation shapes the deployed model’s behavior through training; retrieval-augmented generation instead supplies dynamic knowledge at inference time. Quantization can be combined with distillation to further support production-scale deployment. section 5.4.1 section 1.4.1
