Distillation
Knowledge distillation is a training technique in which a student model learns to reproduce or approximate the outputs or behavior of a teacher model.
LLMs in Production
Distillation is a model-training technique in which a student model learns to reproduce behavior or outputs from a teacher model.
- How it is used
- Uses a smaller student to reduce deployment size, latency, and resource demands while retaining useful teacher performance.
- How deep it goes
- Explains teacher selection, student training, prediction-matching loss, and trade-offs, but gives no dependable rule for student size.
- Examples
- Uses BERT-to-DistilBERT finetuning and names Llama 2 70B and Falcon 180B as possible teachers.
- What you can do after
- Apply it via a teacher–student training workflow, then compare student size and task performance against the deployment goal.
Build a Reasoning Model (From Scratch)
A model adaptation approach that trains a smaller student model to imitate a larger teacher model’s outputs or behavior.
- How it is used
- Uses teacher-generated reasoning traces to turn a general pretrained model into a smaller, more efficient reasoning model.
- How deep it goes
- Explains hard and soft objectives, dataset preparation, training-loop mechanics, resource trade-offs, and checkpoint evaluation.
- Examples
- Demonstrates DeepSeek-R1 supervision, Qwen3 0.6B students, MATH training problems, MATH-500 evaluation, checkpoints, and loss curves.
- What you can do after
- Build a distilled student by generating teacher data, fine-tuning with cross-entropy, monitoring validation loss, and evaluating checkpoints.
Read how Build a Reasoning Model (From Scratch) explains distillation →
Reinforcement Learning from Human Feedback
Distillation is a model-training technique in which a student model learns to reproduce the outputs or behavior of a teacher model.
- How it is used
- Supplies synthetic supervision for post-training and transfers teacher capabilities into smaller language models.
- How deep it goes
- Derives per-token and sequence-level objectives, including cross-entropy and forward-KL interpretations, while noting sequence matching is approximated by teacher outputs.
- Examples
- Uses teacher-generated completions, preference pairs, filtered datasets, and reasoning or coding skill transfer as concrete pipelines.
- What you can do after
- Apply it via supervised fine-tuning by curating prompts, generating and filtering teacher outputs, then training a student on the resulting supervision.
Read how Reinforcement Learning from Human Feedback explains distillation →
Building Reliable AI Systems
A training process that transfers the performance of a larger model to a smaller model.
- How it is used
- Makes resource-heavy LLM behavior deployable through a smaller student when teacher inference is too slow or expensive.
- How deep it goes
- Details sequence-level prompt–response collection, student fine-tuning, and evaluation without requiring teacher logits or temperature scaling.
- Examples
- Uses e-commerce returns and exchange questions, training pairs, fine-tuning jobs, and accuracy and similarity evaluations.
- What you can do after
- Apply it via OpenAI’s fine-tuning API to train a smaller support model and evaluate its responses before deployment.
Read how Building Reliable AI Systems explains distillation →
Generative AI in Action
Knowledge distillation is a model-compression technique in which a smaller student model learns to reproduce knowledge or behavior from a larger teacher model.
- How it is used
- Supports efficient small language models for real-time, resource-constrained, and lower-cost deployments.
- How deep it goes
- Positions distillation as a compression technique but does not explain its training procedure or mathematical mechanism.
- Examples
- Uses analysis, translation, and summarization as representative small-model language tasks.
- What you can do after
- Evaluate distillation as an approach for trading model scale and quality against deployment speed, cost, and resource requirements.
Transformers in Action
Distillation is a model-compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.
- How it is used
- Compresses large language models for affordable, lower-resource training and inference on commodity GPUs, edge devices, and other constrained hardware.
- How deep it goes
- Explains teacher-student roles, softened targets, and ground-truth supervision, but does not specify the training loss or schedule.
- Examples
- Uses DistilBERT, Llama 4 variants, GPT-5 mini, Claude Sonnet, Gemini 1.5 Flash, and Gemma 3 270M as distilled-model examples.
- What you can do after
- Evaluate/understand how smaller student models trade model scale for efficiency while retaining teacher capabilities and behavior.
Related concepts
- alignment trainingcontrasts with · Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.
- teacher modeluses · A teacher model is a model whose outputs, labels, or learned behavior provide training signals or targets for another model.
- student modelproduces · A student model is a model trained to approximate or reproduce the outputs, capabilities, or behavior of another model that provides training guidance.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.