liveBook · Manning

Distillation

As explained in AI Model Evaluation

A model compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.

Distillation is a model optimization technique for reducing inference cost and latency. It is especially relevant when a model’s size or complexity makes prediction speed a product concern. section 4.2.2 section 4.4.2

Use distillation when evaluating the production trade-offs around:

  • Model inference latency, which varies with architecture, parameter count, input complexity, infrastructure, and routing. section 4.2.2
  • Application load time, where delayed predictions can slow personalized content and reduce user engagement. section 4.4.2
  • Batching versus live inference: reducing model cost does not remove the need to choose between throughput and responsiveness. section 4.2.2

Recommendation experience

A recommendation application can use distillation to reduce the time required to generate recommendation lists. Validate the result through model inference time and full application load time, because faster model predictions matter only when users receive a responsive experience. section 4.4.2 section 4.2.2

How AI Model Evaluation covers it

How deep it goes
Identifies distillation as a latency optimization but does not explain its teacher-student training process or implementation steps.
Examples
Uses model predictions, recommendation lists, application load time, batch inference, and live inference as production contexts.
What it assumes
Assumes familiarity with model inference latency, model size, application load time, and production deployment trade-offs.
What you can do after
Apply it as one option for reducing model size and inference time, then validate the resulting application responsiveness.
How it connects
Connects distillation with quantization, pruning, inference latency, batching, live inference, and hardware accelerators.

Also explained in

Compare how 14 books explain distillation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.