Distillation
Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
Rearchitecting LLMs
Knowledge distillation is a model-compression technique in which a smaller student model learns to reproduce useful behavior from a larger teacher model.
- How it is used
- Recovers capabilities after structural pruning while retaining a smaller student architecture.
- How deep it goes
- Develops training loops, dataset preparation, loss combinations, layer mapping, and controlled recovery comparisons.
- Examples
- Compares pruned Gemma recovery on SlimPajama and Cosmopedia and a tool-call specialist using masked completion labels.
- What you can do after
- Builds a distillation training loop and applies OptiPFair to recover and evaluate pruned models.
Build a DeepSeek Model (From Scratch)
Knowledge distillation is a model-training technique in which a student model learns to reproduce information or behavior provided by a teacher model.
- How it is used
- Uses distillation as post-training compression to transfer reasoning behavior from a costly teacher into more practical student models.
- How deep it goes
- Develops temperature-scaled soft-target loss and a runnable training loop, then contrasts them with offline reasoning-trace fine-tuning.
- Examples
- Compares hard-label and distilled toy students and reports DeepSeek-R1’s Qwen and Llama reasoning benchmarks.
- What you can do after
- Build a classical distillation pipeline from scratch in PyTorch and evaluate the trade-offs of reasoning-trace distillation.
Read how Build a DeepSeek Model (From Scratch) explains distillation →
LLM Customization and Fine-Tuning
Training a student model to imitate the behavior of a teacher model, often a larger or stronger one.
- How it is used
- Transfers task-specific teacher behavior into a cheaper student for enterprise inference.
- How deep it goes
- Walks through teacher-data generation, filtering, student training, and three-way evaluation, with black-box and white-box tradeoffs.
- Examples
- Uses an IT support assistant and DeepSeek-R1 student benchmarks to illustrate behavioral and task-dependent retention.
- What you can do after
- Builds a black-box student pipeline and evaluates its retention and safety against base and teacher models.
Read how LLM Customization and Fine-Tuning explains distillation →
LLMs in Production
Distillation is a model-training technique in which a student model learns to reproduce behavior or outputs from a teacher model.
- How it is used
- Uses distillation to make capable language models more practical under resource and response-time constraints.
- How deep it goes
- Explains teacher and student selection, prediction-matching loss, task labels, and subsequent finetuning, while noting that student sizing lacks a settled recipe.
- Examples
- Illustrates distillation with Alpaca’s LLaMA 7B finetuning and a BERT-to-DistilBERT example.
- What you can do after
- Evaluate distillation as a compression option by weighing student performance and deployment efficiency against training and maintenance costs.
Build a Reasoning Model (From Scratch)
A model adaptation approach that trains a smaller student model to imitate a larger teacher model’s outputs or behavior.
- How it is used
- Uses teacher-generated reasoning traces to turn a general pretrained model into a smaller, more efficient reasoning model.
- How deep it goes
- Explains hard and soft objectives, dataset preparation, training-loop mechanics, resource trade-offs, and checkpoint evaluation.
- Examples
- Demonstrates DeepSeek-R1 supervision, Qwen3 0.6B students, MATH training problems, MATH-500 evaluation, checkpoints, and loss curves.
- What you can do after
- Build a distilled student by generating teacher data, fine-tuning with cross-entropy, monitoring validation loss, and evaluating checkpoints.
Read how Build a Reasoning Model (From Scratch) explains distillation →
Transfer Learning for Natural Language Processing
Knowledge distillation is a neural-network compression technique that trains a smaller student model to mimic the outputs of a larger teacher model.
- How it is used
- Uses distillation to improve parameter efficiency and make pretrained NLP models easier to deploy.
- How deep it goes
- Explains teacher-derived soft labels and ground-truth losses, then illustrates compression with DistilBERT rather than exhaustively surveying methods.
- Examples
- Uses DistilBERT and reported BERT-to-bi-LSTM compression results to illustrate size and inference-time tradeoffs.
- What you can do after
- Apply it via Hugging Face transformers using DistilBERT to obtain a smaller BERT model.
Read how Transfer Learning for Natural Language Processing explains distillation →
Applied Reinforcement Learning
A model-training technique in which a student model learns to reproduce useful outputs or behavior of a teacher model.
- How it is used
- Uses offline oracle labels to make language-model signals practical within a live reinforcement-learning system.
- How deep it goes
- Explains offline labeling, student fine-tuning, deployment, and fidelity measurement without specifying a training algorithm.
- Examples
- Uses a frontier model to label news-derived market signals for a smaller model in a portfolio manager.
- What you can do after
- Apply it via offline oracle labeling, student fine-tuning, deployment, and measurement of surviving signal.
Read how Applied Reinforcement Learning explains distillation →
LLM Evaluation and Alignment, The Foundational Ideas
Distillation is a model-compression technique in which a smaller model is trained to reproduce the behavior or outputs of a larger model.
- How it is used
- Reduces the cost and latency of production models or evaluation judges while preserving useful teacher behavior.
- How deep it goes
- Explains teacher-student transfer, three signal types, validation, and a two-phase pipeline without fully detailing every training objective.
- Examples
- Uses GPT-4-to-small-judge transfer and critique-and-revise Constitutional AI traces as concrete examples.
- What you can do after
- Apply it by collecting teacher outputs, training a smaller model, validating against a golden set, and monitoring agreement after deployment.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains distillation →
Reinforcement Learning from Human Feedback
Distillation is a model-training technique in which a student model learns to reproduce the outputs or behavior of a teacher model.
- How it is used
- Uses teacher supervision to generate post-training data and transfer skills to smaller language models.
- How deep it goes
- Derives per-token and sequence-level matching objectives and outlines completion-based training pipelines.
- Examples
- Uses mathematical reasoning, coding, and instruction tuning on outputs from an RLVR-trained teacher as skill-transfer examples.
- What you can do after
- Evaluate/understand how teacher distributions or generated sequences supervise a student and why prompt curation and filtering matter.
Read how Reinforcement Learning from Human Feedback explains distillation →
Retrieval Augmented Generation, The Foundational Ideas
Knowledge distillation is a training technique in which a smaller student model learns to reproduce the behavior or outputs of a larger teacher model.
- How it is used
- Transfers a larger model’s retrieval and self-critique behavior into a smaller deployable model.
- How deep it goes
- Distinguishes probability-matching from behavioral distillation and traces annotation, fine-tuning, decoding, and retrieval gating.
- Examples
- Uses Critic-labeled query-answer data, reflection tokens, and Retrieve-token thresholds to demonstrate behavioral transfer.
- What you can do after
- Builds a smaller model that emits learned control signals for grounded generation and adaptive retrieval.
Read how Retrieval Augmented Generation, The Foundational Ideas explains distillation →
Building Reliable AI Systems
A training process that transfers the performance of a larger model to a smaller model.
- How it is used
- Makes a resource-intensive language model’s behavior practical to deploy in a smaller, faster student.
- How deep it goes
- Explains sequence-level distillation from teacher responses through student fine-tuning, contrasting it with matching soft probabilities.
- Examples
- Uses an e-commerce support assistant and customer return conversations to illustrate teacher-generated training data.
- What you can do after
- Apply it via OpenAI’s fine-tuning API using collected teacher responses to train a smaller student.
Read how Building Reliable AI Systems explains distillation →
AI Model Evaluation
A model compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.
- How it is used
- Uses model compression to address inference latency and responsiveness in production AI applications.
- How deep it goes
- Identifies distillation as a latency optimization but does not explain its teacher-student training process or implementation steps.
- Examples
- Uses model predictions, recommendation lists, application load time, batch inference, and live inference as production contexts.
- What you can do after
- Apply it as one option for reducing model size and inference time, then validate the resulting application responsiveness.
Generative AI in Action
Knowledge distillation is a model-compression technique in which a smaller student model learns to reproduce knowledge or behavior from a larger teacher model.
- How it is used
- Supports efficient small language models for real-time, resource-constrained, and lower-cost deployments.
- How deep it goes
- Positions distillation as a compression technique but does not explain its training procedure or mathematical mechanism.
- Examples
- Uses analysis, translation, and summarization as representative small-model language tasks.
- What you can do after
- Evaluate distillation as an approach for trading model scale and quality against deployment speed, cost, and resource requirements.
Transformers in Action
Distillation is a model-compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.
- How it is used
- Compresses large language models for affordable, lower-resource training and inference on commodity GPUs, edge devices, and other constrained hardware.
- How deep it goes
- Explains teacher-student roles, softened targets, and ground-truth supervision, but does not specify the training loss or schedule.
- Examples
- Uses DistilBERT, Llama 4 variants, GPT-5 mini, Claude Sonnet, Gemini 1.5 Flash, and Gemma 3 270M as distilled-model examples.
- What you can do after
- Evaluate/understand how smaller student models trade model scale for efficiency while retaining teacher capabilities and behavior.
Also covered in
- Domain-Specific Small Language Models
- Introduction to Generative AI, Second Edition
- Machines that ThinkIn early access (MEAP)
- Building Specialized AI SystemsIn early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- student modelA student model is a model trained with guidance from a teacher model to approximate its outputs, behavior, or learned capabilities.
- teacher modelA teacher model is a model whose outputs, behavior, or learned representations provide training signals for another model.
Related concepts
- teacher modeluses · A teacher model is a model whose outputs, behavior, or learned representations provide training signals for another model.
- student modeluses · A student model is a model trained with guidance from a teacher model to approximate its outputs, behavior, or learned capabilities.
- alignment trainingcontrasts with · Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.
- supervised fine-tuningkind of · Supervised fine-tuning is further training a pretrained model on labeled input-output examples to adapt its behavior to desired outputs.
Questions about distillation
What is distillation?
Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.
Which Manning books explain distillation?
Rearchitecting LLMs, Build a DeepSeek Model (From Scratch), LLM Customization and Fine-Tuning, LLMs in Production, Build a Reasoning Model (From Scratch), Transfer Learning for Natural Language Processing, Applied Reinforcement Learning, LLM Evaluation and Alignment, The Foundational Ideas and 6 more.
What should I know before learning distillation?
Manning books that teach distillation usually introduce language model, fine-tuning, student model and teacher model first.
What is the difference between distillation and alignment training?
Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model. Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.
What is distillation a kind of?
Distillation is a kind of supervised fine-tuning. Supervised fine-tuning is further training a pretrained model on labeled input-output examples to adapt its behavior to desired outputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.













