Distillation

As explained in LLMs in Production

Distillation is a model-training technique in which a student model learns to reproduce behavior or outputs from a teacher model.

Suppose a large model handles a task well, but is too slow or expensive to deploy. Knowledge distillation trains a smaller student model to mimic a larger teacher model. The goal is to keep much of the teacher’s task performance while reducing model size, response time, and computational cost. section 3.3.1 section 5.3.2

A practical distillation workflow is:

• Choose a pretrained teacher and create or select a smaller student, possibly with fewer layers or reduced dimensionality. section 5.3.2
• Train the student on the task using the teacher’s outputs as a target. A loss term can encourage the student’s predictions to resemble the teacher’s alongside learning from task labels. section 5.3.2
• Check the resulting model’s size and task performance; finetune it further for a specific dataset if needed. section 5.3.2

Smaller is not identical

A BERT-to-DistilBERT finetuning example illustrates the intended trade-off: both size and performance drop, but size drops faster. A student may also fail to acquire some emergent qualities of its foundation-model teacher, so the training dataset needs careful curation. section 5.3.2 section 5.3.2

Figure 5.5 Knowledge distillation allows a smaller model to learn from a foundation model to replicate similar behavior with fewer parameters. The student model does not always learn the emergent qualities of the foundation model, so the dataset must be especially curated. The dotted line indicates a special relationship as the student model becomes the specialized LLM.

The diagram shows the teacher-to-student relationship and the caution that fewer parameters do not guarantee every teacher behavior survives. section 5.3.2

Distillation is not a free compression step. Training a student has substantial upfront cost, and improvements to the teacher must be transferred through later training cycles. There is no dependable recipe for choosing how small the student can be; test that choice against the task. Compared with straightforward post-training quantization, distillation demands more training work but produces a separate, smaller model. section 3.3.1 section 3.5

How LLMs in Production covers it

How deep it goes

Explains teacher selection, student training, prediction-matching loss, and trade-offs, but gives no dependable rule for student size.

Examples

Uses BERT-to-DistilBERT finetuning and names Llama 2 70B and Falcon 180B as possible teachers.

What it assumes

Assumes familiarity with pretrained models, task labels, loss functions, and finetuning while defining the teacher–student relationship.

What you can do after

Apply it via a teacher–student training workflow, then compare student size and task performance against the deployment goal.

How it connects

Connects knowledge distillation to model compression, finetuning, quantization, pruning, and prompt tuning.

liveBookmanning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A model adaptation approach that trains a smaller student model to imitate a larger teacher model’s outputs or behavior.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackDistillation is a model-training technique in which a student model learns to reproduce the outputs or behavior of a teacher model.
• 

  liveBookmanning.com
  Building Reliable AI SystemsA training process that transfers the performance of a larger model to a smaller model.
• 

  liveBookmanning.com
  Generative AI in ActionKnowledge distillation is a model-compression technique in which a smaller student model learns to reproduce knowledge or behavior from a larger teacher model.
• 

  liveBookmanning.com
  Transformers in ActionDistillation is a model-compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.

Compare how 6 books explain distillation →

Related concepts

• alignment trainingcontrasts with · Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.
• teacher modeluses · A teacher model is a model whose outputs, labels, or learned behavior provide training signals or targets for another model.
• student modelproduces · A student model is a model trained to approximate or reproduce the outputs, capabilities, or behavior of another model that provides training guidance.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.