Chapter 7: Knowledge distillation: Capturing frontier model intelligence
Introduces knowledge distillation as training a smaller student on a stronger teacher’s outputs, explains its enterprise appeal, and flags the need to preserve specialized behavior.
After this chapter you will be able to
- Teacher Student Distillation. Explain how a stronger teacher generates training data for a smaller or cheaper student, which is trained with SFT to reproduce the teacher’s behavior without using the teacher at inference time.
- Enterprise Distillation Appeal. Justify why an enterprise might distill a stronger model instead of relying on it for every request, using the section’s cost, privacy, latency, and operational-independence motivations.
- Black-box distillation workflow. Trace how prompts become teacher responses, verified training data, a trained student, and a comparison with the base and teacher models.
- Focused distillation task. Explain why a specific task, such as IT support for password resets, VPN issues, and software installation, is a better distillation target than a general-purpose assistant.
- Production-relevant prompt diversity. Explain why prompts should vary in phrasing, difficulty, edge cases, and adversarial inputs while mostly reflecting production queries. Describe why a small, intentional share can target known failure points.
- Prompt sourcing and expansion. Describe how to build a diverse prompt set from production query logs or, without logs, from deduplicated domain-expert questions expanded with language-model paraphrases. State what happens to the teacher responses after quality filtering.
- Temperature sampling tradeoff. Explain how temperature changes teacher responses and why 0.7 is a reasonable default for generating student training data.
- Best-of-N response selection. Describe why generating several responses per prompt and selecting one can improve teacher data, and explain the different cost implications for local and API teachers.
- Task-matched teacher choice. Describe how to choose a teacher by task family rather than benchmark rank, including the additional generation and evaluation needs of a reasoning teacher.
- Teacher suitability checks. Explain the confidentiality constraint and the small accuracy check recommended before committing to a distillation run.
- Two Layer Quality Verification. Explain why teacher outputs need automated filtering followed by manual spot-checking, and how the spot-check results guide whether to proceed or revise generation.
- Overlap And Duplicate Removal. Distinguish test-set contamination from internal near-duplicates, and describe why both should be removed before training.
- Teacher Outputs as Training Data. Explain how student training reuses the SFT workflow and what changes in the training data: the student learns from the teacher’s answers to prompts rather than human-curated answers.
- LoRA Student Trade-offs. Justify using a LoRA adapter for the student and describe when full SFT might be needed instead.
- Small Dataset Overfitting Checks. Describe how dataset size affects epoch choice and how to recognize and respond to overfitting during student training.
- Student inference sequence. Describe how to run inference with the trained student: load the base model, attach its adapter with PeftModel.from_pretrained, and generate a response.
- Teacher-derived response behavior. Explain what the illustrated student response demonstrates about the teacher-generated training data.
- Quality filter edge-case tests. Describe the four unit-test cases for the quality filter and explain why the tests should change when domain-specific thresholds change.
- Teacher output filter calibration. Explain the suggested starting rejection range and how manual review of rejected outputs guides adjustments to the filter.
- Three-way distillation quality comparison. Explain why distillation evaluation compares the unadapted base model, the teacher, and the student, and describe how to calculate and interpret the student’s retention rate.
- Common Test and Scoring. Explain how the three models are compared using the same held-out test split, Token-F1 as the primary metric, and per-category scores.
- Deterministic Evaluation Decoding. Explain why evaluation uses greedy decoding even though teacher-data generation used a nonzero temperature.
- Three-Way Distillation Result. Explain what the held-out comparison shows about the base model, teacher, and distilled student, including the student's reproduction of the teacher's house format.
- Token-F1 Interpretation Limits. Distinguish what Token-F1 measures from what the behavioral and safety checks must assess.
- Token-F1 interpretation limits. Explain why token overlap can rank models yet misrepresent the usefulness or correctness of an individual student response.
- LLM judge assessment dimensions. Describe what the LLM judge compares and the three qualities it rates when evaluating a distilled student.
- Combined evaluation workflow. Explain how Token-F1 and the LLM judge are used together, including what disagreement between their per-category scores can reveal.
- Teacher Data Scaling Tradeoff. Explain how retention changes as teacher-data volume grows and why generating more examples usually takes priority over tuning learning rate or LoRA rank, especially when moving from 200 to 500 examples.
- Student failure diagnosis. Describe how to inspect the validation examples with the lowest Token-F1, distinguish format failures, content gaps, and hallucinations, and match each to the remedy given in the section.
- Evaluation to generation feedback. Explain how patterns in student failures guide the next round of teacher-data generation.
- Four drivers of distillation value. Explain how replacing a frontier API with a locally deployed distilled student can reduce ongoing cost, limit data sent to third parties, lower latency, and reduce dependence on the API provider.
- Distillation benefits and limits. Explain that distillation transfers behavior a teacher has already learned into a student that is cheaper to serve, rather than making the student inherently smarter. Describe why the student may repeat or worsen the teacher’s mistakes.
- Teacher student training framework. Explain how the teacher produces reference outputs for a set of tasks and the student trains on them using standard supervised learning. Describe why the student can be useful for routine work without matching the teacher on every edge case.
- Transferred behavioral patterns. Distinguish the behavioral patterns transferred in this framework from factual information, giving examples of the response behaviors the section names.
- Teacher Data Versus Human Data. Distinguish distillation from fine-tuning a student on the human-authored data used to train its teacher. Explain that distillation is SFT on teacher-generated data; the section identifies data quality, consistency, and coverage as differences and says human effort is front-loaded rather than eliminated.
- Expanded Training Data Coverage. Explain how a teacher can supply reference responses for synthetic prompts covering cases absent from the original human dataset, extending the student’s effective training distribution.
- Consistent teacher behavior. Explain how consistent teacher outputs help a student learn compared with human-authored outputs that vary in phrasing, reasoning, and edge-case judgments.
- Consistent teacher errors. Explain why a student may reproduce a teacher’s recurring mistakes and why distillation therefore needs quality verification.
- Teacher information and training signal. Distinguish white-box distillation, which trains on the teacher’s token probability distributions, from black-box distillation, which trains on the teacher’s text responses. Explain that black-box training can use standard SFT; its distinguishing feature is teacher-generated supervision rather than a new training algorithm.
- Distillation approach selection. Justify why black-box distillation is the practical enterprise default and describe the circumstances in which white-box distillation is worth choosing.
- Task Difficulty and Retention. Explain how the DeepSeek-R1 results show that retention depends on both task difficulty and student capacity, and how to use an easy-versus-hard split when judging a distillation candidate.
- Reasoning Trace Verification. Describe what a student must learn from distilled reasoning traces and why checking final answers or plausible-looking traces alone is insufficient.
- Teacher Student Compatibility. Justify selecting a student base model with relevant pre-training capability rather than choosing solely by model size or teacher quality.
- Black-Box KL Training Limits. Explain why forward KL is the default for the chapter’s black-box pipeline and why optimizing reverse KL requires white-box access to the teacher.
- Forward and reverse KL penalties. Explain what each divergence penalizes in the student and how that affects the outputs it learns to produce.
- Multimodal output trade-off. Describe when the divergence choice makes a practical difference, using the JSON-versus-Markdown example.
- Selective token supervision. Explain SelecTKD’s finding that emphasizing informative tokens produced larger gains than changing the KL divergence across the whole sequence.
- On-policy distillation findings. Describe how the reported on-policy approaches use student-generated sequences or mistakes, and state their reported benefits and cost.
- Divergence choice and student behavior. Distinguish the behavioral priorities of forward and reverse KL divergence in distillation.
- Enterprise distillation starting point. Justify starting most enterprise distillation projects with standard SFT on teacher outputs, and describe the circumstances that warrant considering alternatives.
- Data Volume Versus Divergence. Explain why adding training examples is usually a higher-impact choice than switching between forward and reverse KL, and identify the tasks where the divergence choice may matter.
- Distillation investment trade-offs. Explain why distillation is not worthwhile in every scenario, naming its upfront work and the factors that affect its return on investment.
- Distillation decision rule. Apply the section’s rule of thumb: when three or more factors favor distillation, it is likely worth the investment; when most favor the teacher API, the API is the simpler and safer choice.
- Privacy and latency constraints. Explain why distillation should not be judged on cost alone: data that cannot leave the network can make it valuable even with a cheap API, while a requirement for responses under 200 ms calls for local inference at scale.
- Task-specific retention targets. Explain why the acceptable retention rate depends on the use case and why the target should be set before results are available.
- Production-based teacher baseline. Describe how to use a small pilot of production queries to set the student’s quality target relative to measured teacher quality.
- Illustrative distillation break-even. Explain how the section compares an upfront distillation investment with ongoing API savings and inference costs. State the illustrative scale and break-even range without treating them as guaranteed.
- Costs beyond simple savings. Describe which costs the headline break-even figures leave out and how existing versus newly provisioned GPU infrastructure changes the comparison.
- Hybrid teacher student routing. Explain how a hybrid deployment uses the distilled student for routine queries and the teacher API for complex or high-stakes queries, and why that division can save money while preserving quality on difficult cases.
- Confidence routing and limits. Describe how the router can decide whether to serve a student response or escalate it, using simple output-based signals, and explain why this does not guarantee correctness.
- Low volume economics. Explain why low query volume can make distillation a poor investment.
- High stakes quality gap. Justify staying with the teacher API when a quality loss would have material consequences.
- Changing task maintenance costs. Explain why rapidly changing tasks can erase distillation's inference savings.
- Distillation project readiness. Explain what a distillation project needs beyond a financial case: a domain expert to spot-check teacher outputs, an ML engineer to run training, and a production pipeline able to serve LoRA adapters.
- Baseline and refinement iterations. Describe the planned minimum of two full iterations: establish a measured baseline by generating and filtering teacher data, training, and evaluating; then target weak categories, retrain, check safety, and compare results before deciding whether to deploy or continue. Describe the optional polish iteration if issues remain.
- Distillation provenance and lineage. Trace the documented path from teacher model and prompt set through filtering and student training, and distinguish the student that produces an output from the teacher that shaped its behavior.
- Teacher output usage rights. Explain why terms-of-service and licensing review must precede a teacher-data pipeline, especially when proprietary API outputs will become training data.
- Qualitative distillation trade-off. Explain how the base model, SFT teacher, and distilled LoRA student differ in the VPN example, and why examining their outputs reveals differences Token-F1 cannot capture.
- Deployment fit for distilled students. Distinguish tasks where a distilled student’s preserved structure may be sufficient from tasks where its loss of specificity may be unacceptable.
- Teacher Output Quality Filter. Explain why the quality filter is worth keeping even when teacher outputs usually look clean, and describe the bad examples it rejects.
- Category Performance Versus Example Counts. Explain why the experiment does not support predicting category retention from training-example counts, and state the resulting evaluation practice.
- LoRA Rank Tradeoff. Describe why rank 16 was the experimental sweet spot for this distillation run and when a higher rank might be worth trying.
- Frontier teacher substitution. Describe how a frontier API can replace the local teacher while leaving the rest of the distillation pipeline unchanged.
- Retention versus student quality. Explain why a frontier-taught student may retain a smaller share of its teacher’s capability yet achieve higher overall quality than a student taught by the local SFT model.
- Targeted distillation feedback. Describe how evaluation guides additional frontier-teacher data generation and the improvement the section reports from iterating.
- Safety alignment transfer limits. Explain that a distilled student may produce unsafe content even when trained on benign outputs from a teacher that correctly refuses harmful requests, and that this risk applies to both black-box and white-box distillation.
- Missing refusal behavior. Explain why black-box distillation can preserve task performance without transferring the teacher’s safety alignment, and describe the section’s proposed remedy.
- Refusal regressions and release gate. Explain what regressed in the distilled student and why the result blocks deployment, including the refusal-rate release gate.
- Refusal examples in teacher data. Explain why adding verified teacher refusals to the training data is the preferred response when a distilled student fails to refuse harmful requests.
- Robustness check deployment gate. State how robustness checks govern deployment of a distilled student, including the section’s minimum pass rates.
What it builds on
- Hands-on: SFT training workflow (chapter 6) · The SFT training workflow that distillation reuses.
- Where an adapter beats prompting (chapter 5) · The example of an adapted assistant whose behavior matters beyond generic answers.
- Does your training data reflect your production distribution? (chapter 3) · The importance of reflecting the production distribution in training data.
- Production log mining (chapter 3) · Production logs as a source of task-relevant queries.
- Step 2: Build the prompt with style anchoring (chapter 3) · Using prompts to shape synthetic data generation.
- Golden test sets and held-out evaluation (chapter 3) · Held-out evaluation supplies the test set checked for contamination.
- Data lineage (chapter 3) · Data lineage provides the earlier rationale for tracking how a dataset was produced.
- How LoRA works (chapter 5) · LoRA’s parameter-efficient adapter approach.
Section by section
Introduction
Introduces knowledge distillation as training a smaller student on a stronger teacher’s outputs, explains its enterprise appeal, and flags the need to preserve specialized behavior.
- Teacher Student Distillation: Explain how a stronger teacher generates training data for a smaller or cheaper student, which is trained with SFT to reproduce the teacher’s behavior without using the teacher at inference time.
Why it matters: This is the basic relationship behind the chapter’s teacher-to-student training example. - Enterprise Distillation Appeal: Justify why an enterprise might distill a stronger model instead of relying on it for every request, using the section’s cost, privacy, latency, and operational-independence motivations.
Why it matters: These tradeoffs explain why distillation may be worth considering even when the teacher gives better answers. - Specialized Behavior Preservation: Explain why judging a distilled support assistant requires attention to its house answer format and domain behavior, not only overall accuracy.
Why it matters: A student that gives accurate answers may still fail to behave like the specialized assistant it is meant to replace.
Key terms: knowledge distillation, teacher, student, SFT, cost, privacy, latency, operational independence, house answer format, domain behavior
7.1 Hands-on: The distillation recipe
This section outlines a black-box distillation pipeline and explains why its local demonstration may retain more teacher behavior than distillation from a frontier API.
- Black-box distillation workflow: Trace how prompts become teacher responses, verified training data, a trained student, and a comparison with the base and teacher models.
Why it matters: The student’s quality depends on the prompts, teacher outputs, verification, training, and evaluation—not just the training run. - Teacher source portability: Explain what stays the same when the teacher is local rather than a frontier API, and identify the stage that changes.
Why it matters: You can understand the workflow without treating a frontier API as a requirement. - Demonstration retention caveat: Explain why this demonstration’s student is expected to retain more teacher behavior than a 4B student trained from a frontier API teacher.
Why it matters: The demonstration’s retention result should not be assumed for a much stronger frontier teacher.
Key terms: black-box pipeline, teacher data, automated filters, student, standard SFT, local model, frontier API, Stage 2, teacher data generation, Qwen3-4B
7.1.1 Stage 1: Task identification and prompt design
This section explains how to define a focused distillation task and design diverse, production-relevant prompts to send to a teacher model.
- Focused distillation task: Explain why a specific task, such as IT support for password resets, VPN issues, and software installation, is a better distillation target than a general-purpose assistant.
Why it matters: A clear task boundary makes the teacher examples more useful to the student. - Production-relevant prompt diversity: Explain why prompts should vary in phrasing, difficulty, edge cases, and adversarial inputs while mostly reflecting production queries. Describe why a small, intentional share can target known failure points.
Why it matters: Varied prompts help the student cope with real queries and known weak spots rather than memorizing repeated patterns. - Prompt sourcing and expansion: Describe how to build a diverse prompt set from production query logs or, without logs, from deduplicated domain-expert questions expanded with language-model paraphrases. State what happens to the teacher responses after quality filtering.
Why it matters: A small set of authentic questions can become a practical, varied source of teacher examples.
Key terms: distillation, teacher examples, student’s retention rate, prompt diversity, production distribution, edge cases, difficulty variations, adversarial inputs, refusal behavior, production query logs
7.1.2 Stage 2: Teacher data generation
This section explains how to generate useful teacher responses by choosing sampling settings, managing cost and response length, and selecting and vetting a teacher suited to the task.
- Temperature sampling tradeoff: Explain how temperature changes teacher responses and why 0.7 is a reasonable default for generating student training data.
Why it matters: Sampling settings shape both the range and the quality of examples the student sees. - Best-of-N response selection: Describe why generating several responses per prompt and selecting one can improve teacher data, and explain the different cost implications for local and API teachers.
Why it matters: Selecting among candidates can improve training examples, but the generation budget determines how many candidates are practical. - Batching API teacher requests: Explain why the section recommends generating frontier-API teacher data in a batch.
Why it matters: Batch discounts can make an API-generated training set cheaper. - Length limits for teacher responses: Explain why longer-answer tasks should allow a higher generation token limit and filter responses by length afterward.
Why it matters: An apparently usable but truncated teacher answer can teach the student to stop before completing a response. - Task-matched teacher choice: Describe how to choose a teacher by task family rather than benchmark rank, including the additional generation and evaluation needs of a reasoning teacher.
Why it matters: The student learns from the teacher's responses, so teacher fit matters more than a general leaderboard position. - Teacher suitability checks: Explain the confidentiality constraint and the small accuracy check recommended before committing to a distillation run.
Why it matters: Checking the teacher first can prevent both unauthorized data transfer and an unpromising distillation run.
Key terms: temperature sampling, greedy decoding, degenerate responses, best-of-N, local teacher, API teacher, frontier API, batch processing, max_new_tokens, truncated responses
7.1.3 Stage 3: Quality verification
This section explains how to filter and verify teacher-generated data, prevent overlap and duplication, and record the resulting dataset before student training.
- Two Layer Quality Verification: Explain why teacher outputs need automated filtering followed by manual spot-checking, and how the spot-check results guide whether to proceed or revise generation.
Why it matters: A student trained on unchecked teacher answers can inherit mistakes that simple format filters cannot detect. - Overlap And Duplicate Removal: Distinguish test-set contamination from internal near-duplicates, and describe why both should be removed before training.
Why it matters: Removing both kinds of overlap protects the credibility of evaluation and the variety of behavior the student sees. - Filter Rejection Diagnosis: Describe how to respond when automated filtering rejects an unexpectedly large share of teacher outputs.
Why it matters: Investigating rejected examples prevents genuine teacher-output problems from being hidden by weaker filters. - Category Mix After Filtering: Explain why the category distribution of retained teacher examples matters for the student's overall quality.
Why it matters: A single large category can dominate an overall result even if smaller categories behave differently. - Distillation Manifest Reproducibility: Explain what the manifest records and why distillation needs records of generation, filtering, and splitting settings.
Why it matters: The manifest lets a team trace a student result back through the choices that shaped its training data.
Key terms: automated filtering, manual spot-checking, cross-model verification, LLM-as-a-judge, test-set contamination, internal near-duplicates, TF-IDF cosine similarity, rejection rate, temperature, system prompts
7.1.4 Stage 4: Student training
Student distillation reuses the SFT training workflow with verified teacher outputs, while LoRA offers a compact student adapter whose training still needs careful overfitting checks.
- Teacher Outputs as Training Data: Explain how student training reuses the SFT workflow and what changes in the training data: the student learns from the teacher’s answers to prompts rather than human-curated answers.
Why it matters: It identifies what makes this training run distillation rather than an ordinary SFT run. - LoRA Student Trade-offs: Justify using a LoRA adapter for the student and describe when full SFT might be needed instead.
Why it matters: The adapter choice affects deployment, updates, and how much teacher behavior the student can capture. - Small Dataset Overfitting Checks: Describe how dataset size affects epoch choice and how to recognize and respond to overfitting during student training.
Why it matters: A falling training loss alone can hide a student that is memorizing teacher examples. - Training Signals Versus Quality: Distinguish a healthy student training run from evidence that the student has matched or surpassed the teacher.
Why it matters: It prevents a good-looking training dashboard from being mistaken for proof of teacher-level performance.
Key terms: SFTTrainer, chat-template formatting, teacher-generated data, student, LoRA adapter, full SFT, hot-swapping, regularizer, capacity, epoch count
7.1.5 Student inference
This subsection shows how to run a trained distillation student and what teacher-derived behavior to look for in its response.
- Student inference sequence: Describe how to run inference with the trained student: load the base model, attach its adapter with PeftModel.from_pretrained, and generate a response.
Why it matters: The student can be run using the familiar LoRA inference pattern. - Teacher-derived response behavior: Explain what the illustrated student response demonstrates about the teacher-generated training data.
Why it matters: These response traits show the kind of teacher behavior the student has picked up.
Key terms: student model, base model, adapter, PeftModel.from_pretrained, teacher’s behavioral patterns, teacher-generated training data, student
7.1.6 Testing the quality filter
This section explains how to test and calibrate the teacher-output quality filter so it rejects bad examples without discarding useful training data.
- Quality filter edge-case tests: Describe the four unit-test cases for the quality filter and explain why the tests should change when domain-specific thresholds change.
Why it matters: The tests help catch filter changes that would let poor teacher outputs into training or throw away useful ones. - Teacher output filter calibration: Explain the suggested starting rejection range and how manual review of rejected outputs guides adjustments to the filter.
Why it matters: Reviewing actual rejections keeps a numerical target from overriding the quality of the training data.
Key terms: quality filter, unit tests, degenerate repetition, CI pipeline, starting calibration, rejection rate, teacher outputs, rejected outputs
7.2 Evaluating distillation quality
This section defines how to compare a distilled student with its teacher and an unadapted base model, using retention rate as the key measure.
- Three-way distillation quality comparison: Explain why distillation evaluation compares the unadapted base model, the teacher, and the student, and describe how to calculate and interpret the student’s retention rate.
Why it matters: Retention rate shows how much of the teacher’s measured performance the student preserves.
Key terms: base model, teacher, student, distillation retention rate, Token-F1
7.2.1 Three-way comparison
This section explains how to compare three models on the same held-out data using consistent scoring and reproducible decoding.
- Common Test and Scoring: Explain how the three models are compared using the same held-out test split, Token-F1 as the primary metric, and per-category scores.
Why it matters: A single overall score can hide the categories where the student struggles. - Deterministic Evaluation Decoding: Explain why evaluation uses greedy decoding even though teacher-data generation used a nonzero temperature.
Why it matters: Separating diverse data generation from reproducible evaluation makes score differences easier to interpret.
Key terms: held-out test split, Token-F1, per-category scoring, deterministic decoding, do_sample=False, greedy decoding, temperature=0.7, sampling noise
7.2.2 Interpreting the results
This section interprets a three-way distillation evaluation by comparing the student with the base model and teacher, checking behavior beyond Token-F1, and treating category-level retention cautiously.
- Three-Way Distillation Result: Explain what the held-out comparison shows about the base model, teacher, and distilled student, including the student's reproduction of the teacher's house format.
Why it matters: The strongest result is cheap reproduction of the teacher's observed behavior, not just a close word-overlap score. - Token-F1 Interpretation Limits: Distinguish what Token-F1 measures from what the behavioral and safety checks must assess.
Why it matters: A close overlap score alone cannot show that the student answers appropriately or is safe to deploy. - Category Retention And Uncertainty: Describe how to interpret category-level retention and use it when considering tasks for distillation.
Why it matters: The overall retention figure can hide task differences, while a tiny category sample can exaggerate them.
Key terms: Token-F1, teacher, distilled student, house format, LoRA adapter, lexical overlap, format adherence, behavioral safety, retention rate, per-category breakdown
7.2.3 Beyond Token-F1: Using an LLM judge for distillation quality
The section explains why Token-F1 needs a qualitative LLM-judge check when evaluating whether a distilled student’s answers are as useful as its teacher’s.
- Token-F1 interpretation limits: Explain why token overlap can rank models yet misrepresent the usefulness or correctness of an individual student response.
Why it matters: A high overlap score alone cannot tell you whether the student gave a good answer. - LLM judge assessment dimensions: Describe what the LLM judge compares and the three qualities it rates when evaluating a distilled student.
Why it matters: These ratings check answer qualities that shared-token counts cannot capture. - Combined evaluation workflow: Explain how Token-F1 and the LLM judge are used together, including what disagreement between their per-category scores can reveal.
Why it matters: Using both measures helps distinguish real improvements from answers that merely look similar to the teacher’s.
Key terms: Token-F1, token-level overlap, relative ranking signal, LLM judge, correctness, completeness, coherence, checkpoint, final candidate model, per-category scores
7.2.4 Scaling behavior
This section explains how teacher-data volume affects distillation retention and how to use the approximate scaling results to prioritize additional data over hyperparameter tuning.
- Teacher Data Scaling Tradeoff: Explain how retention changes as teacher-data volume grows and why generating more examples usually takes priority over tuning learning rate or LoRA rank, especially when moving from 200 to 500 examples.
Why it matters: It helps you decide whether to spend effort generating teacher examples or tuning the student. - Scaling Estimate Limitations: Distinguish the published results and working estimates behind the scaling figures, and explain why the chapter's 159-example result should not be treated as typical frontier-API scaling.
Why it matters: It prevents you from treating approximate figures or a special-case result as guaranteed outcomes.
Key terms: retention rate, teacher data, learning rate, LoRA rank, published results, working estimates, frontier-API teacher, student capacity
7.2.5 Error analysis
This section shows how to diagnose a distillation student's weak validation results and use the findings to improve training data or choose another remedy.
- Student failure diagnosis: Describe how to inspect the validation examples with the lowest Token-F1, distinguish format failures, content gaps, and hallucinations, and match each to the remedy given in the section.
Why it matters: Diagnosing the kind of failure helps you avoid tuning the wrong part of the pipeline. - Evaluation to generation feedback: Explain how patterns in student failures guide the next round of teacher-data generation.
Why it matters: Evaluation findings can tell you what teacher examples to generate next. - Low retention data priority: State the section's recommended first intervention when student retention falls below 70%, and explain why it takes priority over hyperparameter tuning.
Why it matters: This rule of thumb helps prioritize a practical response to poor retention.
Key terms: Token-F1, format failures, content gaps, hallucinations, LoRA rank, RAG, teacher data, data generation stage, iterative refinement loop, retention rate
7.3 Why distillation matters
This section explains the business case for replacing a frontier API with a distilled student, why API access makes distillation practical, and why domain-specific evaluation still matters.
- Four drivers of distillation value: Explain how replacing a frontier API with a locally deployed distilled student can reduce ongoing cost, limit data sent to third parties, lower latency, and reduce dependence on the API provider.
Why it matters: These trade-offs help determine whether building a student is preferable to continuing to buy API access. - API access and distillation practicality: Explain why frontier LLM APIs have made distillation more accessible than when teams needed to train their own large teacher models.
Why it matters: Access to an existing teacher changes which teams can realistically attempt distillation. - Benchmark results and domain quality: Describe what the cited industrial results suggest about distilled-student performance, and explain why domain-specific evaluation remains important.
Why it matters: A strong general benchmark result alone does not establish that a student meets a particular deployment need.
Key terms: frontier API, distilled student, data egress risk, operational independence, frontier LLM APIs, teacher model, knowledge distillation, standard benchmarks, domain-specific evaluation
7.3.1 What distillation can and cannot do
This section explains that distillation makes a teacher’s task-specific behavior cheaper to serve, while preserving its limits, and introduces multiple teachers as a way to reduce dependence on one teacher’s blind spots.
- Distillation benefits and limits: Explain that distillation transfers behavior a teacher has already learned into a student that is cheaper to serve, rather than making the student inherently smarter. Describe why the student may repeat or worsen the teacher’s mistakes.
Why it matters: It sets realistic expectations for both deployment cost and student quality. - Multiple teacher contributions: Describe how using training data from several teachers can bring their different strengths into one student rather than relying on one teacher’s blind spots. Distinguish the stated black-box approach from the white-box difficulty.
Why it matters: It explains a way to reduce reliance on one teacher’s strengths and weaknesses.
Key terms: distillation, teacher, student, task-specific behavioral patterns, inference, multi-teacher distillation, black-box teachers, white-box teachers, tokenizers
7.4 Distillation intuition
7.4.1 The teacher-student framework
This section explains how a student model learns a teacher’s task-specific behavior from reference outputs and why that can be useful for routine work.
- Teacher student training framework: Explain how the teacher produces reference outputs for a set of tasks and the student trains on them using standard supervised learning. Describe why the student can be useful for routine work without matching the teacher on every edge case.
Why it matters: It explains why a less capable student can still be practical to deploy. - Transferred behavioral patterns: Distinguish the behavioral patterns transferred in this framework from factual information, giving examples of the response behaviors the section names.
Why it matters: It clarifies what the student is learning from the teacher’s examples.
Key terms: teacher, student, reference outputs, standard supervised learning, behavioral patterns, structure responses, reasoning steps, express uncertainty
7.4.2 Distillation vs. fine-tuning on the same data
This section contrasts distillation with training on the teacher’s original human-authored data and explains how teacher-generated responses can make training economical while expanding coverage.
- Teacher Data Versus Human Data: Distinguish distillation from fine-tuning a student on the human-authored data used to train its teacher. Explain that distillation is SFT on teacher-generated data; the section identifies data quality, consistency, and coverage as differences and says human effort is front-loaded rather than eliminated.
Why it matters: This distinction explains why teacher-generated training data can be economical at scale without eliminating human work. - Expanded Training Data Coverage: Explain how a teacher can supply reference responses for synthetic prompts covering cases absent from the original human dataset, extending the student’s effective training distribution.
Why it matters: The teacher can provide training examples for cases that would be costly to annotate by hand.
Key terms: distillation, SFT on teacher data, human-authored data, data quality, consistency, coverage, synthetic prompts, edge cases, rare scenarios, adversarial inputs
7.4.3 What the student actually learns
This section explains why consistent teacher outputs can help a student learn a task, and why the same consistency can pass on the teacher’s errors.
- Consistent teacher behavior: Explain how consistent teacher outputs help a student learn compared with human-authored outputs that vary in phrasing, reasoning, and edge-case judgments.
Why it matters: It explains the learning advantage this section attributes to teacher-generated outputs. - Consistent teacher errors: Explain why a student may reproduce a teacher’s recurring mistakes and why distillation therefore needs quality verification.
Why it matters: A consistent training signal is useful only if its recurring errors are checked.
Key terms: teacher outputs, human-authored outputs, student, coherent behavioral pattern, quality verification, human-authored training data
7.4.4 White-box vs. black-box distillation
This section contrasts white-box and black-box distillation by the teacher information available to the student, explains when each is useful, and distinguishes self-distillation from transferring knowledge between models.
- Teacher information and training signal: Distinguish white-box distillation, which trains on the teacher’s token probability distributions, from black-box distillation, which trains on the teacher’s text responses. Explain that black-box training can use standard SFT; its distinguishing feature is teacher-generated supervision rather than a new training algorithm.
Why it matters: The information available from the teacher determines what signal the student can learn from. - Distillation approach selection: Justify why black-box distillation is the practical enterprise default and describe the circumstances in which white-box distillation is worth choosing.
Why it matters: The choice depends on both the quality sought and whether the teacher’s internal probabilities are accessible. - Self-distillation’s distinct purpose: Describe how one model can serve as both teacher and student in self-distillation, and distinguish its purpose from compression-focused transfer between models.
Why it matters: Self-distillation solves a different adaptation problem from making a separate student imitate a teacher.
Key terms: white-box distillation, black-box distillation, logits, teacher’s text outputs, SFT, open-weight models, proprietary APIs, quality target, reasoning model, operational simplicity
7.4.5 The DeepSeek-R1 distillation results
DeepSeek-R1 distillation results show how task difficulty, student capacity, reasoning-trace quality, and model compatibility affect what a smaller student retains.
- Task Difficulty and Retention: Explain how the DeepSeek-R1 results show that retention depends on both task difficulty and student capacity, and how to use an easy-versus-hard split when judging a distillation candidate.
Why it matters: An overall retention score can hide failures on precisely the hard cases an enterprise needs the student to handle. - Reasoning Trace Verification: Describe what a student must learn from distilled reasoning traces and why checking final answers or plausible-looking traces alone is insufficient.
Why it matters: A convincing explanation is not enough evidence that the student's reasoning produced its answer. - Teacher Student Compatibility: Justify selecting a student base model with relevant pre-training capability rather than choosing solely by model size or teacher quality.
Why it matters: Choosing models for their fit to the task can matter as much as choosing the strongest available teacher. - Black Box Starting Baseline: Explain why the section recommends starting with black-box distillation even when teacher weights are available, and when to consider white-box distillation.
Why it matters: A simpler baseline lets you establish whether the added demands of white-box distillation are warranted.
Key terms: retention, pass@1, MATH-500, AIME 2024, student capacity, reasoning trace distillation, chain-of-thought reasoning traces, reasoning faithfulness, reasoning-answer coherence, base model selection
7.5 Reverse KL divergence for selective knowledge transfer
This section explains what KL divergence measures and why reverse-KL training requires teacher access that the chapter’s black-box distillation pipeline does not have.
- Black-Box KL Training Limits: Explain why forward KL is the default for the chapter’s black-box pipeline and why optimizing reverse KL requires white-box access to the teacher.
Why it matters: It distinguishes what the chapter’s pipeline can actually train on from a technique that needs greater access to the teacher. - KL Divergence Basics: Describe what KL divergence measures and state its asymmetry and zero-value condition.
Why it matters: This gives the reader the meaning of the measure used to discuss matching a teacher’s output distribution.
Key terms: black-box, white-box access, sampled outputs, cross-entropy loss, forward KL, reverse KL, Kullback-Leibler divergence, DKL(P || Q), reference distribution, asymmetric
7.5.1 Forward vs. reverse KL
This section contrasts how forward and reverse KL shape a distilled student, qualifies the classical intuition, and explains when the choice matters.
- Forward and reverse KL penalties: Explain what each divergence penalizes in the student and how that affects the outputs it learns to produce.
Why it matters: The different penalties explain why one student may cover more teacher behaviors while another produces more focused outputs. - Limits of classical KL intuition: Distinguish the classical mean-seeking and mode-seeking intuition from the more qualified finding reported for LLM knowledge distillation.
Why it matters: This qualification keeps a useful analogy from becoming an overconfident prediction about LLM training. - Multimodal output trade-off: Describe when the divergence choice makes a practical difference, using the JSON-versus-Markdown example.
Why it matters: For a deployment requiring valid JSON, consistent output can be more valuable than retaining both formats.
Key terms: forward KL divergence, reverse KL divergence, teacher’s distribution, mean-seeking, mode-seeking, head vs. tail focus, precision/recall trade-off, multimodal, JSON, Markdown
7.5.2 Recent findings
This subsection reports findings on adapting KL divergence, targeting informative tokens, and using student-generated sequences in knowledge distillation.
- Adaptive KL blending: Describe how Adaptive KL blends forward and reverse KL during training according to local characteristics of the teacher’s distribution.
Why it matters: It shows that the divergence choice can adapt during training rather than remain fixed. - Selective token supervision: Explain SelecTKD’s finding that emphasizing informative tokens produced larger gains than changing the KL divergence across the whole sequence.
Why it matters: It directs attention to which tokens receive training emphasis, not just the divergence chosen. - On-policy distillation findings: Describe how the reported on-policy approaches use student-generated sequences or mistakes, and state their reported benefits and cost.
Why it matters: It connects potential quality and robustness gains to the extra compute required by on-policy distillation.
Key terms: Adaptive KL, AKL, divergences, teacher’s distribution, SelecTKD, informative tokens, loss, forward KL, reverse KL, on-policy reverse KL distillation
7.5.3 Summary of divergence approaches
This section contrasts how forward and reverse KL shape a distilled student’s behavior and recommends when enterprise projects should consider alternatives to standard SFT.
- Divergence choice and student behavior: Distinguish the behavioral priorities of forward and reverse KL divergence in distillation.
Why it matters: The divergence objective can change what kinds of answers the student tends to produce. - Enterprise distillation starting point: Justify starting most enterprise distillation projects with standard SFT on teacher outputs, and describe the circumstances that warrant considering alternatives.
Why it matters: It gives you a practical default without ruling out a different objective when the task or results call for one.
Key terms: forward KL, reverse KL, distributional coverage, recall, mode concentration, decisiveness, standard SFT on teacher outputs, selective token weighting, retention rate, on-policy reverse KL
7.5.4 Where this matters in practice
This subsection compares the practical payoff of changing KL divergence with adding training data and explains why newer adaptive methods are not yet practical for most teams.
- Data Volume Versus Divergence: Explain why adding training examples is usually a higher-impact choice than switching between forward and reverse KL, and identify the tasks where the divergence choice may matter.
Why it matters: It helps you prioritize the change likely to produce the larger quality improvement. - Adaptive Method Adoption Limits: Explain why promising adaptive divergence approaches are currently impractical for most teams.
Why it matters: Promising experimental results do not by themselves make a technique economical to adopt.
Key terms: forward KL, reverse KL, Token-F1, data-volume lever, divergence lever, AKL, selective token weighting, TRL, SFTTrainer, custom training loops
7.6 When distillation is worth it (and when it isn’t)
This section frames distillation as an investment whose value depends on costs, quality requirements, deployment scale, and operational constraints.
- Distillation investment trade-offs: Explain why distillation is not worthwhile in every scenario, naming its upfront work and the factors that affect its return on investment.
Why it matters: These trade-offs help you decide whether a distillation project is worth pursuing at all.
Key terms: distillation, upfront cost, ROI, deployment scale, acceptable quality gap, operational constraints
7.6.1 Decision framework
This section gives a six-factor rule of thumb for choosing distillation over a teacher API and explains why privacy and latency can outweigh cost.
- Distillation decision rule: Apply the section’s rule of thumb: when three or more factors favor distillation, it is likely worth the investment; when most favor the teacher API, the API is the simpler and safer choice.
Why it matters: The rule helps you make a structured choice rather than defaulting to distillation or an API. - Privacy and latency constraints: Explain why distillation should not be judged on cost alone: data that cannot leave the network can make it valuable even with a cheap API, while a requirement for responses under 200 ms calls for local inference at scale.
Why it matters: A cheap API may still be unsuitable when data-location or response-time requirements rule it out.
Key terms: distillation, teacher API, six factors, privacy, latency, local inference
7.6.2 Quality targets
This section explains how to set a distilled student’s quality target before training, using task-specific error costs and measured teacher performance.
- Task-specific retention targets: Explain why the acceptable retention rate depends on the use case and why the target should be set before results are available.
Why it matters: An upfront, task-appropriate target makes it harder to excuse a disappointing result after the fact. - Production-based teacher baseline: Describe how to use a small pilot of production queries to set the student’s quality target relative to measured teacher quality.
Why it matters: A production-based baseline ties the target to what the teacher actually achieves on your task.
Key terms: retention rate, task criticality, error costs, hindsight bias, production queries, teacher quality, student target
7.6.3 Cost model
This section uses an illustrative cost model to show when distilling a frontier API teacher may pay off and which costs can change that calculation.
- Illustrative distillation break-even: Explain how the section compares an upfront distillation investment with ongoing API savings and inference costs. State the illustrative scale and break-even range without treating them as guaranteed.
Why it matters: It gives you a way to compare a one-time investment with recurring inference spending. - Costs beyond simple savings: Describe which costs the headline break-even figures leave out and how existing versus newly provisioned GPU infrastructure changes the comparison.
Why it matters: A project that looks cheap from API savings alone may cost more once people and infrastructure are included. - Token pricing sensitivity: Explain why current API prices can make projected distillation savings unreliable and how to estimate API spending from expected usage.
Why it matters: Using request counts or assuming API prices stay fixed can distort the payback estimate.
Key terms: break-even, upfront investment, ongoing API costs, total cost of ownership, engineering time, training infrastructure, managed-platform improvements, GPU infrastructure, amortize, API pricing
7.6.4 The middle ground: Hybrid deployment
This section explains how hybrid deployment routes routine queries to a distilled student and harder cases to a teacher API, balancing cost, quality, and monitoring.
- Hybrid teacher student routing: Explain how a hybrid deployment uses the distilled student for routine queries and the teacher API for complex or high-stakes queries, and why that division can save money while preserving quality on difficult cases.
Why it matters: You can reserve expensive teacher calls for cases where their quality matters most. - Confidence routing and limits: Describe how the router can decide whether to serve a student response or escalate it, using simple output-based signals, and explain why this does not guarantee correctness.
Why it matters: Simple routing can limit exposure to visibly uncertain answers, but it cannot catch every wrong answer. - Teacher escalation monitoring signal: Explain what a rising share of queries sent to the teacher may signal over time.
Why it matters: Routing rates can alert a team that the student is handling fewer queries confidently than before.
Key terms: teacher API, distilled student, hybrid pattern, long tail, confidence-based router, hedging phrases, output token probabilities, fraction of queries routed to the teacher, distribution drift, model degradation
7.6.5 When to skip distillation
The section identifies three situations in which distillation is unlikely to be worth its cost or quality trade-off.
- Low volume economics: Explain why low query volume can make distillation a poor investment.
Why it matters: A cheaper per-query student is not necessarily cheaper overall. - High stakes quality gap: Justify staying with the teacher API when a quality loss would have material consequences.
Why it matters: Inference savings may not justify consequential errors. - Changing task maintenance costs: Explain why rapidly changing tasks can erase distillation's inference savings.
Why it matters: Serving savings matter only if the student can be maintained economically.
Key terms: low volume, API costs, upfront distillation investment, unacceptable quality gap, teacher API, high-stakes, rapidly evolving tasks, teacher data, retraining the student, inference savings
7.6.6 Planning a distillation project
The section explains the people, production capability, iterations, and time assumptions needed to plan a distillation project.
- Distillation project readiness: Explain what a distillation project needs beyond a financial case: a domain expert to spot-check teacher outputs, an ML engineer to run training, and a production pipeline able to serve LoRA adapters.
Why it matters: A project can look financially attractive but still lack the people or serving capability needed to complete it. - Baseline and refinement iterations: Describe the planned minimum of two full iterations: establish a measured baseline by generating and filtering teacher data, training, and evaluating; then target weak categories, retrain, check safety, and compare results before deciding whether to deploy or continue. Describe the optional polish iteration if issues remain.
Why it matters: A baseline and a measured refinement pass make the deploy-or-iterate decision more defensible. - Teacher selection time allowance: State the assumption behind the suggested timeline and the extra time to allow if a teacher API still needs evaluation and selection.
Why it matters: A schedule that omits teacher selection can underestimate the time required.
Key terms: organizational readiness, domain expert, teacher outputs, LoRA adapters, baseline retention rate, weakest categories, quality filter thresholds, safety robustness check, teacher API, API testing
7.6.7 Distillation and regulatory compliance
This section explains how to document a distilled model for regulatory scrutiny, check whether teacher outputs may legally be used for training, and weigh the upfront investment against expected savings.
- Distillation provenance and lineage: Trace the documented path from teacher model and prompt set through filtering and student training, and distinguish the student that produces an output from the teacher that shaped its behavior.
Why it matters: These records make it possible to explain a production model's origins and outputs to auditors. - Teacher output usage rights: Explain why terms-of-service and licensing review must precede a teacher-data pipeline, especially when proprietary API outputs will become training data.
Why it matters: Checking usage rights early avoids building a pipeline that cannot legally be used. - Distillation investment conditions: State the section's economic rule of thumb for when distillation can pay off.
Why it matters: The compliance work sits alongside a practical decision about whether the investment is worthwhile.
Key terms: data provenance, model lineage, manifest, model metadata, Terms of service (ToS), licensing, teacher-data pipeline, proprietary API outputs, capital investment, query volumes
7.7 Practical lessons from the pipeline
Introduces three perspectives that the pipeline’s quantitative results do not capture: qualitative model differences, practical experimental findings, and changes needed for a frontier API teacher.
7.7.1 Qualitative comparison
The section compares base, teacher, and distilled-student outputs to show what distillation preserves, what it loses, and when that trade-off may be acceptable.
- Qualitative distillation trade-off: Explain how the base model, SFT teacher, and distilled LoRA student differ in the VPN example, and why examining their outputs reveals differences Token-F1 cannot capture.
Why it matters: Looking at actual answers shows useful differences that a token-overlap score can miss. - Deployment fit for distilled students: Distinguish tasks where a distilled student’s preserved structure may be sufficient from tasks where its loss of specificity may be unacceptable.
Why it matters: The same loss of detail can be tolerable in one application and disqualifying in another.
Key terms: Token-F1, teacher (SFT model), student (distilled LoRA), lossy compression, distilled students, structure and format, specificity gap
7.7.2 What worked and what did not
This section draws three practical lessons from the distillation experiments: filter teacher outputs, evaluate performance by category, and choose LoRA rank based on observed results.
- Teacher Output Quality Filter: Explain why the quality filter is worth keeping even when teacher outputs usually look clean, and describe the bad examples it rejects.
Why it matters: A single bad teacher response can become a behavior the student is trained to imitate. - Category Performance Versus Example Counts: Explain why the experiment does not support predicting category retention from training-example counts, and state the resulting evaluation practice.
Why it matters: Per-category evaluation can reveal outcomes that training-set counts alone would not predict. - LoRA Rank Tradeoff: Describe why rank 16 was the experimental sweet spot for this distillation run and when a higher rank might be worth trying.
Why it matters: The rank choice should reflect measured retention and the capacity needed to bridge the teacher–student gap.
Key terms: quality filter, length bounds, degenerate-repetition test, category balance, retention, test split, LoRA rank, retention rate, behavioral gap, adapter
7.7.3 Extrapolating to frontier API teachers
This section explains how to use a frontier API as a distillation teacher, why retention may fall even as student quality rises, and how targeted iteration can improve results.
- Frontier teacher substitution: Describe how a frontier API can replace the local teacher while leaving the rest of the distillation pipeline unchanged.
Why it matters: You can change the source of teacher responses without redesigning the student-training pipeline. - Retention versus student quality: Explain why a frontier-taught student may retain a smaller share of its teacher’s capability yet achieve higher overall quality than a student taught by the local SFT model.
Why it matters: Retention relative to the teacher and the student’s overall quality are different measures of success. - Targeted distillation feedback: Describe how evaluation guides additional frontier-teacher data generation and the improvement the section reports from iterating.
Why it matters: Evaluation tells you where additional teacher calls are most useful, rather than merely producing more data indiscriminately.
Key terms: frontier API, JSONL, quality filters, LoRA student, retention rate, capability gap, frontier teacher, 4B student, feedback loop, lowest-retention categories
7.8 Student model robustness checks
This section warns that distillation does not reliably transfer a teacher model’s safety alignment to its student.
- Safety alignment transfer limits: Explain that a distilled student may produce unsafe content even when trained on benign outputs from a teacher that correctly refuses harmful requests, and that this risk applies to both black-box and white-box distillation.
Why it matters: The teacher’s safety behavior cannot be assumed to carry over to the student.
Key terms: safety alignment, distillation, black-box, white-box, student, teacher
7.8.1 Why safety does not transfer
This section explains why a distilled student may lose the teacher’s safety behavior and why distillation data needs explicit refusal examples and safety checks.
- Missing refusal behavior: Explain why black-box distillation can preserve task performance without transferring the teacher’s safety alignment, and describe the section’s proposed remedy.
Why it matters: A student that performs its task well may still answer harmful requests that its teacher would refuse. - Safety-aware output checks: Explain how a teacher’s borderline outputs can affect the student and why ordinary quality filtering is insufficient.
Why it matters: An answer can look like high-quality training data while still teaching the student a harmful pattern.
Key terms: safety alignment, black-box methods, refusal examples, distillation data, borderline content, quality filtering, safety-aware spot-checking, Stage 3 (quality verification)
7.8.2 Robustness check implementation
This section shows how to check a distilled student's safety against its base model, interpret checker artifacts, and block deployment when harmful-request refusals regress.
- Distilled student robustness check: Describe how the safety regression check is adapted for a distilled student: load the student as a LoRA adapter and compare its results with the base model's.
Why it matters: Comparing the student with its base reveals safety behavior lost during distillation. - Keyword checker false failures: Explain why the base model's two reported failures in this example are checker artifacts rather than genuine safety gaps.
Why it matters: A reported failure needs interpretation before it is treated as evidence of unsafe behavior. - Refusal regressions and release gate: Explain what regressed in the distilled student and why the result blocks deployment, including the refusal-rate release gate.
Why it matters: A student that answers harmful requests its base refused must not be released just because the distillation pipeline ran successfully.
Key terms: safety regression, distilled student, LoRA adapter, base model, keyword-based checker, general-knowledge check, medical-uncertainty prompt, hedging phrase, harmful-request refusals, refusal-rate drop
7.8.3 Responding to safety regressions
This section explains how to address safety regressions in a distilled student and use robustness checks to decide whether it is ready for deployment.
- Refusal examples in teacher data: Explain why adding verified teacher refusals to the training data is the preferred response when a distilled student fails to refuse harmful requests.
Why it matters: This addresses the missing training behavior rather than only blocking its effects. - Post-distillation safety mitigations: Distinguish safety-specific fine-tuning from inference-time filtering as responses to a safety regression.
Why it matters: The two options differ in whether they change the model or guard what reaches users. - Robustness check deployment gate: State how robustness checks govern deployment of a distilled student, including the section’s minimum pass rates.
Why it matters: A safe teacher does not guarantee a safe student, so deployment needs its own pass criteria.
Key terms: teacher data, refusal examples, (harmful prompt, teacher refusal) pair, safety-specific fine-tuning, LoRA adapter, input/output safety classifier, inference time, robustness check, deployment gate, minimum pass rates
7.9 Summary
7.10 References
This section lists research references related to knowledge distillation without explaining their findings or methods.
Where this is used later
- Model drift: Understanding and responding (chapter 9) · Connects changes in production routing rates to the later discussion of model drift.
- What to version (chapter 9) · Identifying what to version for a production model.
- Production prompt monitoring (chapter 9) · Independent safety testing of distilled students.
- When fine-tuning plateaus (chapter 8) · The finding that safety does not transfer through distillation.
- Security and abuse monitoring (chapter 9) · Safety alignment does not transfer through distillation.
Concepts explained in this chapter
- distillationTraining a student model to imitate the behavior of a teacher model, often a larger or stronger one.
- capability retentionThe student model's score divided by the teacher model's score on the same evaluation metric and set.
- student modelA student model is a smaller model trained to mimic a teacher model during distillation.
- quality filterA screening process that removes low-quality examples before they are used for training.
- teacher modelA model whose outputs or behavior provide targets for training a student model during distillation.
- token-F1An F1 score based on token-level precision and recall between a predicted text and a reference text.
- LLM-as-a-judgeAn evaluation setup in which a language model judges other model outputs against specified criteria.
- operational independenceThe ability to operate a model without dependence on an external provider or its service availability, limits, or policies.
- automated filterThe use of programmatic checks to identify and remove examples that fail specified quality criteria.
- verificationThe process of checking generated training data for correctness, usefulness, and suitability before training.
- low-rank adaptationA learned, low-rank additive change to a model's weights.
- Kullback–Leibler divergenceA measure of how one probability distribution differs from a reference probability distribution.
- reverse KL divergenceThe Kullback–Leibler divergence from a student distribution to a teacher distribution.
- evaluation datasetA subset of examples reserved for evaluating a model rather than updating its parameters.
- refusal behaviorA model's act of declining to fulfill a request, often because complying would be inappropriate or unsafe.
- spot-checkingHuman review of a sample of examples to identify quality problems that automated checks may miss.
- latencyThe time between sending a request and receiving a response, including network and processing delays.
- privacyThe requirement that sensitive data not be exposed to external systems during model use or deployment.
- alignment trainingSteering a model's behavior toward human or organizational preferences and constraints.
- promptThe input supplied to a model, potentially containing instructions, context, and examples.
- prompt engineeringSteering a model's behavior through its input instructions or examples without updating its weights.
- provenanceThe record of data’s origins, transformations, and movement through processing systems.
- LoRA rankThe rank hyperparameter that sets the dimensionality of a LoRA adapter’s low-rank updates.
- break-even pointThe point at which two alternatives have equal total cost under the given assumptions.
- costThe money required to train, operate, or maintain a model.
- qualitative assessmentAssessment of model outputs by examining their characteristics rather than relying only on numerical scores.
- safety regression suiteThe practice of repeatedly evaluating model changes for newly introduced safety failures.
- refusal exampleA refusal example is a training instance that demonstrates declining a harmful or disallowed request.
- cosine similarityA measure of the angle-based similarity between two vectors, often used to compare weighted text representations.
- reasoning modelA model optimized to perform multi-step reasoning tasks, especially those requiring structured deliberation.
- accuracyThe proportion of evaluated predictions or answers that are correct.
- data driftA change over time in the characteristics or distribution of data a model encounters, potentially reducing its performance.
- refusal rateRefusal rate is the proportion of tested requests that a model refuses, with the request set determining whether it measures appropriate or harmful-request refusals.
- data qualityThe degree to which adaptation data is correct, relevant, and representative enough to support reliable model behavior.
- greedy decodingGreedy decoding is a generation method that selects the highest-probability next token at each step.
- recallA measure of how much of the probability mass or set of outcomes represented by a teacher distribution is captured by a student distribution.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
