Distillation is a model optimization technique for reducing inference cost and latency. It is especially relevant when a model’s size or complexity makes prediction speed a product concern. section 4.2.2 section 4.4.2
Use distillation when evaluating the production trade-offs around:
- Model inference latency, which varies with architecture, parameter count, input complexity, infrastructure, and routing. section 4.2.2
- Application load time, where delayed predictions can slow personalized content and reduce user engagement. section 4.4.2
- Batching versus live inference: reducing model cost does not remove the need to choose between throughput and responsiveness. section 4.2.2
Recommendation experience
A recommendation application can use distillation to reduce the time required to generate recommendation lists. Validate the result through model inference time and full application load time, because faster model predictions matter only when users receive a responsive experience. section 4.4.2 section 4.2.2













