Knowledge distillation compresses capabilities from a larger teacher model into a smaller student model. It helps small language models deliver useful language performance with lower resource, training, and operating costs. section 2.6
Distillation is especially valuable when deployment constraints matter:
- Faster training and inference support real-time applications and systems with limited processing capacity. section 2.6
- Smaller models can extend deployment to edge devices and organizations with limited budgets. section 2.6
- Distillation can support analysis, translation, and summarization, but smaller models may lose quality on complex tasks. section 2.6
