Chapter 6: Knowledge recovery through distillation
The introduction explains why pruning makes knowledge recovery necessary and why recovery begins with the choice of what to remove.
After this chapter you will be able to
- Pruning and knowledge loss. Explain why knowledge recovery matters after depth or width pruning: pruning tends to remove model knowledge, and greater pruning tends to cause greater loss. Describe recovery as the stage that determines whether the smaller model can match or surpass the base model.
- Pruning choice and recovery. Explain that recovery depends on both what was removed and how the model is trained afterward. Describe the comparison in which importance-based block pruning recovered performance in less training time than removing the last blocks, despite removing the same number.
- Experimental pruning choice. Justify testing block-selection strategies before distillation rather than choosing one solely because it has theoretical support. Explain how the choice affects recovery time and the capacity the model can regain.
- Controlled pruning comparison. Explain why the experiment removes four blocks from an 18-block model and initially compares strategies using 2,000 Cosmopedia records.
- Four block selection strategies. Distinguish the two importance-based selection strategies from the two position-based heuristics, including which blocks each heuristic removes.
- Balanced subset mixture. Explain how and why the experiment mixes Cosmopedia subsets for recovery.
- Honest validation split. Justify keeping training and validation examples separate when measuring recovery.
- Purpose-matched evaluation metrics. Explain why the comparison uses perplexity alongside capability benchmarks, and why a different application might need different metrics.
- Two-stage candidate comparison. Describe how the four strategies are compared immediately after pruning and after a small knowledge-distillation recovery test, and identify why EXP01 is selected.
- Purpose of feature alignment. Explain why matching only the teacher’s and student’s final outputs is insufficient when the aim is to preserve intermediate representations, and describe feature alignment as matching the results of intermediate Transformer blocks.
- Three block mapping strategies. Distinguish uniform, last-block, and original indices mapping for aligning a 14-block student with an 18-block teacher, including what part of the teacher each strategy preserves or emphasizes.
- Three sources of recovery. Explain what hard labels, teacher logits, and hidden states each contribute to student training, and why the section calls for combining them in a weighted composite function.
- Three-signal compound loss. Explain how the compound loss combines dataset supervision, teacher prediction distributions, and intermediate representation alignment, with alpha, beta, and gamma weighting the three signals.
- Next-token task alignment. Describe how task loss uses dataset labels to preserve next-token prediction, including why logits and labels are shifted and why padding is ignored.
- Teacher distribution matching. Explain what logit loss transfers from the teacher and why its KL-divergence calculation uses student log probabilities and teacher probabilities.
- Mapped hidden-state alignment. Describe how hidden loss compares mapped student and teacher block outputs using cosine direction, and how reshaping makes each token representation a row for comparison.
- Objective-dependent loss weighting. Justify the 0.4/0.4/0.2 baseline and describe how to rebalance hidden-loss weight for representation-focused versus purely generative models.
- Teacher–student training cycle. Trace how each batch passes through the teacher and student, how the compound loss compares their outputs, and why only the student's weights are updated.
- Memory-efficient gradient accumulation. Describe how the loop accumulates gradients to obtain a larger effective batch on limited GPU memory, including loss scaling and the final incomplete group.
- Recovery dependence on data. Explain what the 2K, 15K, and 40K training results show about the amount of data needed for knowledge recovery.
- Feature-alignment strategy trade-offs. Distinguish labels-only training from feature alignment in the reported results, and explain when aligning hidden states may be worth its cost.
- Advanced distillation technique roles. Distinguish what Skew KL Divergence and Feature Dynamics Distillation contribute: Skew KLD changes the teacher–student logit comparison when their probability distributions are far apart, while FDD captures how internal representations change between blocks.
- Data-dependent recovery trade-off. Explain how the value of advanced KD changes with dataset size in the Gemma-3-270m experiments, including its perplexity benefit, capability trade-off, and the cheaper option when data is limited.
- Skewed teacher student target. Describe how Skew KLD changes the comparison target and why its effect changes as the student improves.
- Feature dynamics alignment. Explain how Feature Dynamics Distillation differs from layer-by-layer feature alignment: it computes changes between consecutive layers for both student and teacher, then aligns those changes using cosine similarity.
- Data scale and distillation value. Explain how the value of FDD and Skew KLD changes as the recovery dataset grows: they offer no measurable advantage at 2K samples, a modest perplexity advantage at 15K, and a larger advantage at 40K with little difference in capability retention.
- Context-dependent recovery decisions. Explain why choosing a knowledge-recovery approach requires analysis and experiments for the specific use case rather than a fixed formula.
- Feature alignment weight. Explain when to reduce, disable, or increase the feature-alignment weight gamma, including the computational trade-off.
- Hard and soft label balance. Distinguish the roles of alpha and beta and describe how dataset quality changes their recommended balance.
- Teacher distribution temperature. Explain how temperature changes the teacher's probability distribution and choose a range based on how concentrated it already is.
- Data-dependent technique choice. Explain when the section recommends basic distillation versus activating both advanced techniques, using dataset size, perplexity, and computational cost.
- Width pruning and alignment. Explain why a width-pruned student can still be aligned with its teacher, and describe how width pruning, depth pruning, and knowledge distillation can be combined.
- Production pruning and distillation. Explain how the Minitron and Ministral 3 papers support structural pruning followed by distillation as a practical way to create high-performance small language models, and how distillation-based retraining compares with ordinary fine-tuning of a pruned model.
- Hard and soft label balance. Distinguish the hard-label task loss from the teacher-derived soft-label logits loss, and explain why the chapter retains both even though the cited Mistral result favors pure logits distillation.
- Layer removal choices. Distinguish removing the last N layers from using activation analysis to select layers based on their measured contribution.
- Teacher student layer mapping. Describe the uniform, last, and original teacher–student layer mappings and when uniform versus last mapping is presented as useful.
- Conditional value of feature alignment. Explain why feature alignment may add little with moderate pruning and abundant data but help with aggressive compression or limited data.
- Advanced distillation under constraints. Explain why Skew KLD and Feature Dynamics Distillation can be useful when compression is high or the domain is specialized, especially without enterprise-scale data.
- Limits of advanced losses. Explain why advanced distillation does not remove the need for quality recovery data and sound pruning choices.
What it builds on
- How depth pruning works (chapter 4) · Depth pruning removes Transformer blocks.
- Identifying neurons for removal (chapter 5) · Width pruning removes neurons.
- Choosing the blocks to discard (chapter 4) · Choosing which Transformer blocks to discard.
- Recovering knowledge (chapter 2) · Recovering knowledge after pruning.
- Static block selection (chapter 4) · Static block selection was introduced earlier.
- Data-Driven block selection (chapter 4) · Data-driven block selection was introduced earlier.
- Understanding cosine similarity (chapter 4) · Cosine similarity as a way to assess representation changes across blocks.
- Reconstructing the MLP module (chapter 5) · Reconstructing an MLP after removing neurons.
Section by section
Introduction
The introduction explains why pruning makes knowledge recovery necessary and why recovery begins with the choice of what to remove.
- Pruning and knowledge loss: Explain why knowledge recovery matters after depth or width pruning: pruning tends to remove model knowledge, and greater pruning tends to cause greater loss. Describe recovery as the stage that determines whether the smaller model can match or surpass the base model.
Why it matters: A smaller model's usefulness depends on how well it regains capabilities lost through pruning. - Pruning choice and recovery: Explain that recovery depends on both what was removed and how the model is trained afterward. Describe the comparison in which importance-based block pruning recovered performance in less training time than removing the last blocks, despite removing the same number.
Why it matters: Choosing what to retain can change the training effort needed to recover performance.
Key terms: structural pruning, depth, width, knowledge recovery, base model, importance-based pruning, Transformer blocks, Knowledge Distillation, training time
6.1 Choosing what to prune
This section explains why block-selection strategies should be compared before spending resources on distillation-based recovery.
- Experimental pruning choice: Justify testing block-selection strategies before distillation rather than choosing one solely because it has theoretical support. Explain how the choice affects recovery time and the capacity the model can regain.
Why it matters: Choosing blocks before an expensive recovery run can save GPU time and preserve more recoverable capability. - Block selection approaches: Distinguish data-driven and static depth-pruning choices, and describe what the protection hypothesis recommends preserving.
Why it matters: These are the competing block-selection ideas the reader needs to recognize when comparing pruning candidates.
Key terms: Knowledge Distillation, critical knowledge, recovery ceiling, data-driven methods, calibration data, static methods, protection hypothesis, final Transformer blocks
6.1.1 Experiment configuration
This section sets up a controlled comparison of four ways to remove Transformer blocks before knowledge recovery.
- Controlled pruning comparison: Explain why the experiment removes four blocks from an 18-block model and initially compares strategies using 2,000 Cosmopedia records.
Why it matters: Keeping the pruning amount constant makes the selection strategies comparable. - Four block selection strategies: Distinguish the two importance-based selection strategies from the two position-based heuristics, including which blocks each heuristic removes.
Why it matters: The comparison tests measured block importance against two different assumptions about the final blocks. - Recovery dataset dual purpose: Explain why the dataset matters both for data-driven block selection and for subsequent knowledge-distillation training.
Why it matters: Dataset choice affects both which blocks appear removable and what the recovery effort emphasizes.
Key terms: Transformer blocks, Cosmopedia dataset, knowledge recovery, EXP01, EXP02, EXP03, EXP04, data-driven individual, data-driven consecutive, data-driven strategies
6.1.2 Preparing the recovery dataset
This section explains how to prepare a varied, tokenized recovery dataset and a separate validation set for evaluating knowledge distillation.
- Recovery data quality: Explain why the experiment uses curated synthetic text for knowledge recovery.
Why it matters: The examples used for recovery can help or hinder the student model. - Balanced subset mixture: Explain how and why the experiment mixes Cosmopedia subsets for recovery.
Why it matters: A varied sample helps test recovery of general capabilities, not just performance on one kind of text. - Teacher and text preparation: Describe why the teacher is frozen and why the text is tokenized in batches.
Why it matters: These preparation choices make the recovery data usable without spending resources updating the teacher. - Honest validation split: Justify keeping training and validation examples separate when measuring recovery.
Why it matters: A held-out set makes comparisons of recovery results more trustworthy.
Key terms: Knowledge Distillation, Cosmopedia, synthetic dataset, thematic subsets, balanced dataset, general capabilities, teacher model, freeze, batch tokenization, numerical tensors
6.1.3 Selecting the best candidate
This section selects a pruning strategy by comparing fluency and capability before and after a small distillation test, then identifies a training pitfall when copying the student model.
- Purpose-matched evaluation metrics: Explain why the comparison uses perplexity alongside capability benchmarks, and why a different application might need different metrics.
Why it matters: A pruning strategy is only as good as the capabilities its evaluation actually checks. - Two-stage candidate comparison: Describe how the four strategies are compared immediately after pruning and after a small knowledge-distillation recovery test, and identify why EXP01 is selected.
Why it matters: Checking both stages avoids choosing a candidate solely on its initial damage or its improvement during recovery. - Recovery gain interpretation: Explain why EXP01 can have the smallest recovery gain yet the best final result.
Why it matters: The biggest improvement is not necessarily the best recovered model. - Copied student gradient state: Explain why a student made with deepcopy must not retain the teacher’s frozen gradient state during training.
Why it matters: A copied student can appear ready for recovery training while silently failing to learn.
Key terms: perplexity (PPL), generalist model, benchmarks, EXP01, Knowledge Distillation, cosine similarity analysis, capability retention, Recovery gain, baseline, deepcopy
6.2 Aligning Teacher and Student
The section explains why feature alignment requires matching intermediate blocks of a pruned student to a deeper teacher and compares three ways to make those matches.
- Purpose of feature alignment: Explain why matching only the teacher’s and student’s final outputs is insufficient when the aim is to preserve intermediate representations, and describe feature alignment as matching the results of intermediate Transformer blocks.
Why it matters: It explains why this recovery approach looks inside the models rather than comparing only their answers. - Three block mapping strategies: Distinguish uniform, last-block, and original indices mapping for aligning a 14-block student with an 18-block teacher, including what part of the teacher each strategy preserves or emphasizes.
Why it matters: The mapping determines which teacher representations guide each student block during recovery.
Key terms: classic distillation, hard labels, soft labels, Feature Alignment, intermediate representations, uniform mapping, last-block mapping, original indices mapping, Teacher, Student
6.3 Recovering the knowledge
The section explains why recovering knowledge in a pruned student calls for training with hard labels, teacher logits, and hidden states together.
- Three sources of recovery: Explain what hard labels, teacher logits, and hidden states each contribute to student training, and why the section calls for combining them in a weighted composite function.
Why it matters: These distinct training signals explain what the recovery objective is meant to preserve beyond correct answers alone.
Key terms: hard labels, soft labels, logits, hidden states, composite function
6.3.1 The compound loss function
This section explains how to calculate and balance task, logit, and hidden-state losses to recover a pruned student's knowledge from data and a teacher model.
- Three-signal compound loss: Explain how the compound loss combines dataset supervision, teacher prediction distributions, and intermediate representation alignment, with alpha, beta, and gamma weighting the three signals.
Why it matters: The combined objective determines what the student is encouraged to preserve during recovery. - Next-token task alignment: Describe how task loss uses dataset labels to preserve next-token prediction, including why logits and labels are shifted and why padding is ignored.
Why it matters: Correct alignment lets dataset supervision reinforce actual next-token prediction rather than mismatched or padded positions. - Teacher distribution matching: Explain what logit loss transfers from the teacher and why its KL-divergence calculation uses student log probabilities and teacher probabilities.
Why it matters: Matching confidence across options transfers information that a single correct label cannot express. - Temperature-softened teacher probabilities: Explain how raising temperature softens the teacher's probability distribution and why that makes alternative tokens useful to the student.
Why it matters: Softening exposes useful alternatives that a sharply peaked teacher distribution can obscure. - Mapped hidden-state alignment: Describe how hidden loss compares mapped student and teacher block outputs using cosine direction, and how reshaping makes each token representation a row for comparison.
Why it matters: This signal helps retain useful internal structure without requiring representation vectors to have identical magnitudes. - Objective-dependent loss weighting: Justify the 0.4/0.4/0.2 baseline and describe how to rebalance hidden-loss weight for representation-focused versus purely generative models.
Why it matters: The weights should reflect whether the recovered model's output text or its representations matter most.
Key terms: compound loss, task loss, logit loss, hidden loss, alpha, beta, Cross-Entropy, hard labels, token shifting, ignore_index=-100
6.3.2 Training loop and strategy comparison
This section explains how the student-training loop uses teacher outputs and compares recovery results across data sizes and distillation strategies.
- Teacher–student training cycle: Trace how each batch passes through the teacher and student, how the compound loss compares their outputs, and why only the student's weights are updated.
Why it matters: This is the basic cycle that recovers the pruned student's behavior without training the teacher. - Transformer block output selection: Explain why the loop requests hidden states and removes the first entry before aligning teacher and student representations.
Why it matters: Keeping only block outputs makes the representations being compared correspond to transformer blocks. - Memory-efficient gradient accumulation: Describe how the loop accumulates gradients to obtain a larger effective batch on limited GPU memory, including loss scaling and the final incomplete group.
Why it matters: Accumulation allows an effective batch larger than the GPU can hold at once. - Recovery dependence on data: Explain what the 2K, 15K, and 40K training results show about the amount of data needed for knowledge recovery.
Why it matters: The amount of recovery data can matter more than adding a more elaborate distillation strategy. - Feature-alignment strategy trade-offs: Distinguish labels-only training from feature alignment in the reported results, and explain when aligning hidden states may be worth its cost.
Why it matters: Choosing a distillation strategy depends on the target metric, training cost, and whether hidden representations are themselves useful outputs.
Key terms: train_student, compute_compound_loss, AdamW optimizer, torch.no_grad(), output_hidden_states=True, hidden states, [1:], embedding layer, transformer block, gradient accumulation
6.4 Advanced recovery techniques (FDD/ Skew KLD)
This section introduces two advanced distillation techniques and uses recovery results to show when their added complexity is worthwhile.
- Advanced distillation technique roles: Distinguish what Skew KL Divergence and Feature Dynamics Distillation contribute: Skew KLD changes the teacher–student logit comparison when their probability distributions are far apart, while FDD captures how internal representations change between blocks.
Why it matters: The two techniques target different information that a pruned student may need during recovery. - Data-dependent recovery trade-off: Explain how the value of advanced KD changes with dataset size in the Gemma-3-270m experiments, including its perplexity benefit, capability trade-off, and the cheaper option when data is limited.
Why it matters: Dataset size helps determine whether the extra recovery technique is worth its computational cost.
Key terms: Skew KL Divergence, teacher and student logits, Feature Dynamics Distillation, internal representations, advanced KD, perplexity, capability retention, basic logit distillation
6.4.1 Skew KLD: correcting distribution bias
This subsection explains how Skew KLD softens distillation loss when teacher and student predictions differ sharply, and when that change may be useful.
- Standard KL under large discrepancies: Explain why standard KL Divergence can be problematic when an aggressively pruned student starts with predictions far from the teacher’s.
Why it matters: This explains the training problem Skew KLD is intended to address. - Skewed teacher student target: Describe how Skew KLD changes the comparison target and why its effect changes as the student improves.
Why it matters: The mixed target is the defining change that lets Skew KLD ease early recovery without keeping the target equally soft throughout training. - Skew KLD use cases: Distinguish the situations in which the section favors Skew KLD from those in which standard KL may suffice.
Why it matters: The choice of loss should reflect how degraded the student is and the available training data.
Key terms: Logits Loss, KL Divergence, teacher, student, steep gradient, Skew KLD, skew_alpha, mixed_probs, teacher_probs, student_probs
6.4.2 Feature Dynamics Distillation
This section explains how Feature Dynamics Distillation aligns changes between consecutive student and teacher layers instead of comparing their representations at individual layers.
- Feature dynamics alignment: Explain how Feature Dynamics Distillation differs from layer-by-layer feature alignment: it computes changes between consecutive layers for both student and teacher, then aligns those changes using cosine similarity.
Why it matters: It captures how representations change across layers, which matching individual layer states alone misses.
Key terms: Feature Dynamics Distillation, FDD, deltas, cosine similarity, transformation vectors
6.4.3 Results: when advanced techniques make a difference
This section compares knowledge-recovery results across dataset sizes to show when advanced distillation helps and which capability remains difficult to restore.
- Data scale and distillation value: Explain how the value of FDD and Skew KLD changes as the recovery dataset grows: they offer no measurable advantage at 2K samples, a modest perplexity advantage at 15K, and a larger advantage at 40K with little difference in capability retention.
Why it matters: Dataset size helps determine whether advanced recovery techniques justify their extra complexity. - Incomplete LAMBADA recovery: Describe how all methods improve LAMBADA performance over the pruned model but remain below the teacher, and what the section infers about recovering long-context language modeling after layer removal.
Why it matters: A strong perplexity result does not mean every capability has returned to the teacher's level.
Key terms: FDD, Skew KLD, Advanced KD, perplexity, capability retention, basic logit distillation, LAMBADA, pruned model, teacher, long-context language modeling
6.5 Practical guidelines for knowledge recovery
This introduction explains why knowledge-recovery choices must be tested against the particular model and use case rather than taken from a universal recipe.
- Context-dependent recovery decisions: Explain why choosing a knowledge-recovery approach requires analysis and experiments for the specific use case rather than a fixed formula.
Why it matters: It keeps you from treating one recovery strategy as the right choice for every model and task.
Key terms: knowledge distillation, data, task, LLM, shape, original training
6.5.1 Composite loss configuration
This section gives practical rules for setting feature-alignment weight, hard- and soft-label weights, and temperature in a composite distillation loss.
- Feature alignment weight: Explain when to reduce, disable, or increase the feature-alignment weight gamma, including the computational trade-off.
Why it matters: It helps you avoid paying for feature alignment when the model does not need it. - Hard and soft label balance: Distinguish the roles of alpha and beta and describe how dataset quality changes their recommended balance.
Why it matters: The balance determines whether the student relies more on dataset answers or the teacher's output distribution. - Teacher distribution temperature: Explain how temperature changes the teacher's probability distribution and choose a range based on how concentrated it already is.
Why it matters: A suitable temperature lets the student learn useful alternatives without unnecessarily softening an already smooth distribution.
Key terms: feature alignment, gamma, intermediate representations, alpha, beta, hard labels, soft labels, distillation dataset, temperature, teacher's probability distributions
6.5.2 When to use advanced techniques
This section explains how to configure Skew KLD and FDD and when enough training data makes their additional cost worthwhile.
- Advanced distillation configuration: Describe how skew_alpha controls Skew KLD and delta controls FDD, including how to disable each technique and how to keep the compound-loss weights summing to 1.0.
Why it matters: These settings let you enable or disable the techniques without misconfiguring the compound loss. - Data-dependent technique choice: Explain when the section recommends basic distillation versus activating both advanced techniques, using dataset size, perplexity, and computational cost.
Why it matters: Dataset size helps determine whether the perplexity gain justifies the extra computation.
Key terms: compute_compound_loss_advanced, Skew KLD, FDD, skew_alpha, delta, compound loss, basic distillation, Advanced KD, perplexity, training data
6.5.3 Combining with width pruning
Width pruning can be combined with depth pruning and knowledge distillation because it leaves block input and output dimensions unchanged, while recovery settings should be tuned through validation.
- Width pruning and alignment: Explain why a width-pruned student can still be aligned with its teacher, and describe how width pruning, depth pruning, and knowledge distillation can be combined.
Why it matters: You can make a student smaller in both width and depth without redesigning teacher–student alignment. - Validation guided recovery settings: Describe how to treat the recommended distillation configurations when applying the combined pipeline.
Why it matters: Validation helps you adapt a flexible recovery setup instead of treating suggested settings as universal.
Key terms: width pruning, intermediate_size, hidden_dim, teacher and student alignment, depth pruning, Knowledge Distillation, recommended Knowledge Distillation configurations, validation, advanced techniques, compound loss
6.6 From paper to practice
This section connects the chapter’s pruning-and-distillation pipeline to published efficient-model work and explains a choice between mixed and pure distillation objectives.
- Production pruning and distillation: Explain how the Minitron and Ministral 3 papers support structural pruning followed by distillation as a practical way to create high-performance small language models, and how distillation-based retraining compares with ordinary fine-tuning of a pruned model.
Why it matters: It shows why the recovery pipeline is presented as a practical model-building approach rather than just an academic exercise. - Hard and soft label balance: Distinguish the hard-label task loss from the teacher-derived soft-label logits loss, and explain why the chapter retains both even though the cited Mistral result favors pure logits distillation.
Why it matters: It explains why a useful composite loss need not be treated as the best choice in every setting.
Key terms: structural pruning, distillation, standard fine-tuning, SLMs, hard labels, soft labels, task loss, logits loss, alpha, beta
6.6.1 Layer selection and alignment
The section explains how to choose layers for removal, align the remaining student layers with a teacher, and judge when feature alignment is useful.
- Layer removal choices: Distinguish removing the last N layers from using activation analysis to select layers based on their measured contribution.
Why it matters: The layers removed determine what the student must recover through distillation. - Teacher student layer mapping: Describe the uniform, last, and original teacher–student layer mappings and when uniform versus last mapping is presented as useful.
Why it matters: The degree of pruning affects which teacher–student alignment is appropriate. - Conditional value of feature alignment: Explain why feature alignment may add little with moderate pruning and abundant data but help with aggressive compression or limited data.
Why it matters: It helps determine when the extra alignment signal is worth using.
Key terms: last N layers, activation analysis, data-driven approach, uniform, last, original, moderate compression ratios, aggressive depth pruning, feature alignment, hard or soft labels
6.6.2 Advanced techniques: when data is scarce
This section explains when advanced distillation can help with limited data and why it cannot compensate for poor data or pruning.
- Advanced distillation under constraints: Explain why Skew KLD and Feature Dynamics Distillation can be useful when compression is high or the domain is specialized, especially without enterprise-scale data.
Why it matters: It helps you judge when advanced distillation may be worth using with limited resources. - Limits of advanced losses: Explain why advanced distillation does not remove the need for quality recovery data and sound pruning choices.
Why it matters: It prevents treating a more sophisticated loss as a fix for weak inputs or pruning decisions.
Key terms: Skew KLD, Feature Dynamics Distillation, compression ratio, specialized domains, quality data, solid pruning strategy, noisy, block selection, loss function
6.7 Hands-on lab
6.8 Summary
Where this is used later
- (chapter intro) (chapter 7) · Earlier work on changing a model through training provides context for weight-based adaptation.
- One call in the agent's loop (chapter 12) · The earlier treatment of recovering knowledge through teacher-student training.
- Recovering knowledge (chapter 13) · Skew KLD addresses distribution bias during recovery.
Concepts explained in this chapter
- distillationKnowledge distillation is a model-compression technique in which a smaller student model learns to reproduce useful behavior from a larger teacher model.
- hard labelA hard label is a target representation that identifies the designated correct class or token without encoding alternative probabilities.
- soft targetA soft label is a probability-weighted target distribution that represents graded model predictions rather than a single discrete target.
- compound lossA compound loss function combines multiple loss components, often using configurable weights.
- logit lossLogit loss is a distillation loss that measures disagreement between student and teacher logits or their derived output distributions.
- task lossA task loss is an optimization objective that measures disagreement between model predictions and hard labels for a specified task.
- feature alignmentFeature alignment is a distillation technique that matches intermediate representations produced by corresponding blocks of teacher and student models.
- Skew KLDSkew KL divergence is an asymmetric divergence that compares a probability distribution with a mixture of distributions, and can be used as a distillation objective.
- hidden stateA hidden state is an internal vector representation produced by a neural model layer for the information processed up to that layer.
- intermediate representationAn intermediate representation is a vector-valued activation produced inside a neural model between its input and output layers.
- hidden lossA hidden loss is a loss term that measures disagreement between intermediate representations produced by a teacher and a student model.
- Kullback–Leibler divergenceKullback–Leibler divergence measures how one probability distribution differs from a reference distribution.
- cosine distanceCosine distance measures dissimilarity between vectors based on their directional alignment.
- cross entropy lossCross-entropy is a loss function that measures the discrepancy between a target probability distribution and a predicted probability distribution.
- softmaxSoftmax converts a vector of scores into nonnegative probabilities that sum to one.
- token shiftingToken shifting aligns each input position with the subsequent token that a causal language model must predict.
- perplexityPerplexity is a language-modeling metric that measures how well a model predicts a sequence, with lower values indicating better predictive performance.
- gradient accumulationGradient accumulation is a training technique that sums gradients across multiple batches before updating model parameters.
- labels-only distillationLabels-only distillation trains a student model using the teacher’s output labels without additional intermediate-feature or distribution-matching signals.
- logit-based distillationLogit distillation is a knowledge-distillation method in which a student learns from a teacher model's output scores.
- training loopA training loop is an iterative procedure that computes model outputs, evaluates an error, and updates trainable parameters over batches and iterations.
- FDDFDD is an advanced knowledge-distillation technique that transfers feature-distribution information from a teacher model to a student model.
- feature dynamics distillationFeature dynamics distillation is a knowledge-distillation technique that aligns the changes in intermediate representations across consecutive layers of student and teacher models.
- baselineA quantitative baseline is an objective reference measurement established before interventions to evaluate their effects.
- capability retentionCapability retention is the proportion of a model's evaluated performance preserved after a structural modification such as pruning.
- pruningA pruning strategy is a method for deciding which model components to remove during model compression.
- recoveryA recovery process is a post-restructuring procedure that restores or improves a model’s performance after substantial structural components have been removed.
- block selection strategyA block selection strategy is a method for choosing which transformer blocks to retain or remove during depth pruning.
- student modelA student model is a model trained to reproduce a teacher model's behavior during knowledge distillation.
- teacher modelA teacher model is a model whose outputs or behavior guide the training of a student model during knowledge distillation.
- small language modelA small language model is a language model with comparatively limited parameters and computational requirements, typically optimized for efficient deployment.
- knowledge recoveryKnowledge recovery is a model-rearchitecting phase that restores or preserves model capabilities after structural changes.
- structured pruningStructured pruning is a model-compression technique that removes organized groups of parameters, such as neurons, channels, or layers.
- width pruningWidth pruning is a structured pruning technique that removes selected neurons or channels to reduce a model's width.
- structural modificationAttention removal is a model modification that removes attention sub-layers from selected transformer blocks.
- data-driven methodData-driven pruning selects model components for removal using measurements from representative input data.
- depth pruningDepth pruning is a model-reduction technique that removes complete layers from a neural network, often targeting later layers.
- static pruningStatic pruning selects model components for removal using fixed structural information or parameter statistics rather than runtime activation behavior.
- calibration dataCalibration data is representative input data used to measure model behavior and guide data-dependent model compression decisions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
