liveBook · Manning

Distillation

Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.

How 14 Manning books explain it

Rearchitecting LLMs

Knowledge distillation is a model-compression technique in which a smaller student model learns to reproduce useful behavior from a larger teacher model.

How it is used
Recovers capabilities after structural pruning while retaining a smaller student architecture.
How deep it goes
Develops training loops, dataset preparation, loss combinations, layer mapping, and controlled recovery comparisons.
Examples
Compares pruned Gemma recovery on SlimPajama and Cosmopedia and a tool-call specialist using masked completion labels.
What you can do after
Builds a distillation training loop and applies OptiPFair to recover and evaluate pruned models.

Read how Rearchitecting LLMs explains distillation →

Build a DeepSeek Model (From Scratch)

Knowledge distillation is a model-training technique in which a student model learns to reproduce information or behavior provided by a teacher model.

How it is used
Uses distillation as post-training compression to transfer reasoning behavior from a costly teacher into more practical student models.
How deep it goes
Develops temperature-scaled soft-target loss and a runnable training loop, then contrasts them with offline reasoning-trace fine-tuning.
Examples
Compares hard-label and distilled toy students and reports DeepSeek-R1’s Qwen and Llama reasoning benchmarks.
What you can do after
Build a classical distillation pipeline from scratch in PyTorch and evaluate the trade-offs of reasoning-trace distillation.

Read how Build a DeepSeek Model (From Scratch) explains distillation →

LLM Customization and Fine-Tuning

Training a student model to imitate the behavior of a teacher model, often a larger or stronger one.

How it is used
Transfers task-specific teacher behavior into a cheaper student for enterprise inference.
How deep it goes
Walks through teacher-data generation, filtering, student training, and three-way evaluation, with black-box and white-box tradeoffs.
Examples
Uses an IT support assistant and DeepSeek-R1 student benchmarks to illustrate behavioral and task-dependent retention.
What you can do after
Builds a black-box student pipeline and evaluates its retention and safety against base and teacher models.

Read how LLM Customization and Fine-Tuning explains distillation →

LLMs in Production

Distillation is a model-training technique in which a student model learns to reproduce behavior or outputs from a teacher model.

How it is used
Uses distillation to make capable language models more practical under resource and response-time constraints.
How deep it goes
Explains teacher and student selection, prediction-matching loss, task labels, and subsequent finetuning, while noting that student sizing lacks a settled recipe.
Examples
Illustrates distillation with Alpaca’s LLaMA 7B finetuning and a BERT-to-DistilBERT example.
What you can do after
Evaluate distillation as a compression option by weighing student performance and deployment efficiency against training and maintenance costs.

Read how LLMs in Production explains distillation →

Build a Reasoning Model (From Scratch)

A model adaptation approach that trains a smaller student model to imitate a larger teacher model’s outputs or behavior.

How it is used
Uses teacher-generated reasoning traces to turn a general pretrained model into a smaller, more efficient reasoning model.
How deep it goes
Explains hard and soft objectives, dataset preparation, training-loop mechanics, resource trade-offs, and checkpoint evaluation.
Examples
Demonstrates DeepSeek-R1 supervision, Qwen3 0.6B students, MATH training problems, MATH-500 evaluation, checkpoints, and loss curves.
What you can do after
Build a distilled student by generating teacher data, fine-tuning with cross-entropy, monitoring validation loss, and evaluating checkpoints.

Read how Build a Reasoning Model (From Scratch) explains distillation →

Transfer Learning for Natural Language Processing

Knowledge distillation is a neural-network compression technique that trains a smaller student model to mimic the outputs of a larger teacher model.

How it is used
Uses distillation to improve parameter efficiency and make pretrained NLP models easier to deploy.
How deep it goes
Explains teacher-derived soft labels and ground-truth losses, then illustrates compression with DistilBERT rather than exhaustively surveying methods.
Examples
Uses DistilBERT and reported BERT-to-bi-LSTM compression results to illustrate size and inference-time tradeoffs.
What you can do after
Apply it via Hugging Face transformers using DistilBERT to obtain a smaller BERT model.

Read how Transfer Learning for Natural Language Processing explains distillation →

Applied Reinforcement Learning

A model-training technique in which a student model learns to reproduce useful outputs or behavior of a teacher model.

How it is used
Uses offline oracle labels to make language-model signals practical within a live reinforcement-learning system.
How deep it goes
Explains offline labeling, student fine-tuning, deployment, and fidelity measurement without specifying a training algorithm.
Examples
Uses a frontier model to label news-derived market signals for a smaller model in a portfolio manager.
What you can do after
Apply it via offline oracle labeling, student fine-tuning, deployment, and measurement of surviving signal.

Read how Applied Reinforcement Learning explains distillation →

LLM Evaluation and Alignment, The Foundational Ideas

Distillation is a model-compression technique in which a smaller model is trained to reproduce the behavior or outputs of a larger model.

How it is used
Reduces the cost and latency of production models or evaluation judges while preserving useful teacher behavior.
How deep it goes
Explains teacher-student transfer, three signal types, validation, and a two-phase pipeline without fully detailing every training objective.
Examples
Uses GPT-4-to-small-judge transfer and critique-and-revise Constitutional AI traces as concrete examples.
What you can do after
Apply it by collecting teacher outputs, training a smaller model, validating against a golden set, and monitoring agreement after deployment.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains distillation →

Reinforcement Learning from Human Feedback

Distillation is a model-training technique in which a student model learns to reproduce the outputs or behavior of a teacher model.

How it is used
Uses teacher supervision to generate post-training data and transfer skills to smaller language models.
How deep it goes
Derives per-token and sequence-level matching objectives and outlines completion-based training pipelines.
Examples
Uses mathematical reasoning, coding, and instruction tuning on outputs from an RLVR-trained teacher as skill-transfer examples.
What you can do after
Evaluate/understand how teacher distributions or generated sequences supervise a student and why prompt curation and filtering matter.

Read how Reinforcement Learning from Human Feedback explains distillation →

Retrieval Augmented Generation, The Foundational Ideas

Knowledge distillation is a training technique in which a smaller student model learns to reproduce the behavior or outputs of a larger teacher model.

How it is used
Transfers a larger model’s retrieval and self-critique behavior into a smaller deployable model.
How deep it goes
Distinguishes probability-matching from behavioral distillation and traces annotation, fine-tuning, decoding, and retrieval gating.
Examples
Uses Critic-labeled query-answer data, reflection tokens, and Retrieve-token thresholds to demonstrate behavioral transfer.
What you can do after
Builds a smaller model that emits learned control signals for grounded generation and adaptive retrieval.

Read how Retrieval Augmented Generation, The Foundational Ideas explains distillation →

Building Reliable AI Systems

A training process that transfers the performance of a larger model to a smaller model.

How it is used
Makes a resource-intensive language model’s behavior practical to deploy in a smaller, faster student.
How deep it goes
Explains sequence-level distillation from teacher responses through student fine-tuning, contrasting it with matching soft probabilities.
Examples
Uses an e-commerce support assistant and customer return conversations to illustrate teacher-generated training data.
What you can do after
Apply it via OpenAI’s fine-tuning API using collected teacher responses to train a smaller student.

Read how Building Reliable AI Systems explains distillation →

AI Model Evaluation

A model compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.

How it is used
Uses model compression to address inference latency and responsiveness in production AI applications.
How deep it goes
Identifies distillation as a latency optimization but does not explain its teacher-student training process or implementation steps.
Examples
Uses model predictions, recommendation lists, application load time, batch inference, and live inference as production contexts.
What you can do after
Apply it as one option for reducing model size and inference time, then validate the resulting application responsiveness.

Read how AI Model Evaluation explains distillation →

Generative AI in Action

Knowledge distillation is a model-compression technique in which a smaller student model learns to reproduce knowledge or behavior from a larger teacher model.

How it is used
Supports efficient small language models for real-time, resource-constrained, and lower-cost deployments.
How deep it goes
Positions distillation as a compression technique but does not explain its training procedure or mathematical mechanism.
Examples
Uses analysis, translation, and summarization as representative small-model language tasks.
What you can do after
Evaluate distillation as an approach for trading model scale and quality against deployment speed, cost, and resource requirements.

Read how Generative AI in Action explains distillation →

Transformers in Action

Distillation is a model-compression technique in which a smaller student model learns to reproduce the behavior of a larger teacher model.

How it is used
Compresses large language models for affordable, lower-resource training and inference on commodity GPUs, edge devices, and other constrained hardware.
How deep it goes
Explains teacher-student roles, softened targets, and ground-truth supervision, but does not specify the training loss or schedule.
Examples
Uses DistilBERT, Llama 4 variants, GPT-5 mini, Claude Sonnet, Gemini 1.5 Flash, and Gemma 3 270M as distilled-model examples.
What you can do after
Evaluate/understand how smaller student models trade model scale for efficiency while retaining teacher capabilities and behavior.

Read how Transformers in Action explains distillation →

Also covered in

What to know first

Related concepts

Questions about distillation

What is distillation?

Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model.

Which Manning books explain distillation?

Rearchitecting LLMs, Build a DeepSeek Model (From Scratch), LLM Customization and Fine-Tuning, LLMs in Production, Build a Reasoning Model (From Scratch), Transfer Learning for Natural Language Processing, Applied Reinforcement Learning, LLM Evaluation and Alignment, The Foundational Ideas and 6 more.

What should I know before learning distillation?

Manning books that teach distillation usually introduce language model, fine-tuning, student model and teacher model first.

What is the difference between distillation and alignment training?

Distillation is a training technique in which a student model learns to approximate the outputs or behavior of a teacher model. Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.

What is distillation a kind of?

Distillation is a kind of supervised fine-tuning. Supervised fine-tuning is further training a pretrained model on labeled input-output examples to adapt its behavior to desired outputs.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.