liveBook · Manning

Chapter 6: Knowledge recovery through distillation

What you will learn in chapter 6 of Rearchitecting LLMs

The introduction explains why pruning makes knowledge recovery necessary and why recovery begins with the choice of what to remove.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains why pruning makes knowledge recovery necessary and why recovery begins with the choice of what to remove.

Key terms: structural pruning, depth, width, knowledge recovery, base model, importance-based pruning, Transformer blocks, Knowledge Distillation, training time

6.1 Choosing what to prune

This section explains why block-selection strategies should be compared before spending resources on distillation-based recovery.

Key terms: Knowledge Distillation, critical knowledge, recovery ceiling, data-driven methods, calibration data, static methods, protection hypothesis, final Transformer blocks

6.1.1 Experiment configuration

This section sets up a controlled comparison of four ways to remove Transformer blocks before knowledge recovery.

Key terms: Transformer blocks, Cosmopedia dataset, knowledge recovery, EXP01, EXP02, EXP03, EXP04, data-driven individual, data-driven consecutive, data-driven strategies

6.1.2 Preparing the recovery dataset

This section explains how to prepare a varied, tokenized recovery dataset and a separate validation set for evaluating knowledge distillation.

Key terms: Knowledge Distillation, Cosmopedia, synthetic dataset, thematic subsets, balanced dataset, general capabilities, teacher model, freeze, batch tokenization, numerical tensors

6.1.3 Selecting the best candidate

This section selects a pruning strategy by comparing fluency and capability before and after a small distillation test, then identifies a training pitfall when copying the student model.

Key terms: perplexity (PPL), generalist model, benchmarks, EXP01, Knowledge Distillation, cosine similarity analysis, capability retention, Recovery gain, baseline, deepcopy

6.2 Aligning Teacher and Student

The section explains why feature alignment requires matching intermediate blocks of a pruned student to a deeper teacher and compares three ways to make those matches.

Key terms: classic distillation, hard labels, soft labels, Feature Alignment, intermediate representations, uniform mapping, last-block mapping, original indices mapping, Teacher, Student

6.3 Recovering the knowledge

The section explains why recovering knowledge in a pruned student calls for training with hard labels, teacher logits, and hidden states together.

Key terms: hard labels, soft labels, logits, hidden states, composite function

6.3.1 The compound loss function

This section explains how to calculate and balance task, logit, and hidden-state losses to recover a pruned student's knowledge from data and a teacher model.

Key terms: compound loss, task loss, logit loss, hidden loss, alpha, beta, Cross-Entropy, hard labels, token shifting, ignore_index=-100

6.3.2 Training loop and strategy comparison

This section explains how the student-training loop uses teacher outputs and compares recovery results across data sizes and distillation strategies.

Key terms: train_student, compute_compound_loss, AdamW optimizer, torch.no_grad(), output_hidden_states=True, hidden states, [1:], embedding layer, transformer block, gradient accumulation

6.4 Advanced recovery techniques (FDD/ Skew KLD)

This section introduces two advanced distillation techniques and uses recovery results to show when their added complexity is worthwhile.

Key terms: Skew KL Divergence, teacher and student logits, Feature Dynamics Distillation, internal representations, advanced KD, perplexity, capability retention, basic logit distillation

6.4.1 Skew KLD: correcting distribution bias

This subsection explains how Skew KLD softens distillation loss when teacher and student predictions differ sharply, and when that change may be useful.

Key terms: Logits Loss, KL Divergence, teacher, student, steep gradient, Skew KLD, skew_alpha, mixed_probs, teacher_probs, student_probs

6.4.2 Feature Dynamics Distillation

This section explains how Feature Dynamics Distillation aligns changes between consecutive student and teacher layers instead of comparing their representations at individual layers.

Key terms: Feature Dynamics Distillation, FDD, deltas, cosine similarity, transformation vectors

6.4.3 Results: when advanced techniques make a difference

This section compares knowledge-recovery results across dataset sizes to show when advanced distillation helps and which capability remains difficult to restore.

Key terms: FDD, Skew KLD, Advanced KD, perplexity, capability retention, basic logit distillation, LAMBADA, pruned model, teacher, long-context language modeling

6.5 Practical guidelines for knowledge recovery

This introduction explains why knowledge-recovery choices must be tested against the particular model and use case rather than taken from a universal recipe.

Key terms: knowledge distillation, data, task, LLM, shape, original training

6.5.1 Composite loss configuration

This section gives practical rules for setting feature-alignment weight, hard- and soft-label weights, and temperature in a composite distillation loss.

Key terms: feature alignment, gamma, intermediate representations, alpha, beta, hard labels, soft labels, distillation dataset, temperature, teacher's probability distributions

6.5.2 When to use advanced techniques

This section explains how to configure Skew KLD and FDD and when enough training data makes their additional cost worthwhile.

Key terms: compute_compound_loss_advanced, Skew KLD, FDD, skew_alpha, delta, compound loss, basic distillation, Advanced KD, perplexity, training data

6.5.3 Combining with width pruning

Width pruning can be combined with depth pruning and knowledge distillation because it leaves block input and output dimensions unchanged, while recovery settings should be tuned through validation.

Key terms: width pruning, intermediate_size, hidden_dim, teacher and student alignment, depth pruning, Knowledge Distillation, recommended Knowledge Distillation configurations, validation, advanced techniques, compound loss

6.6 From paper to practice

This section connects the chapter’s pruning-and-distillation pipeline to published efficient-model work and explains a choice between mixed and pure distillation objectives.

Key terms: structural pruning, distillation, standard fine-tuning, SLMs, hard labels, soft labels, task loss, logits loss, alpha, beta

6.6.1 Layer selection and alignment

The section explains how to choose layers for removal, align the remaining student layers with a teacher, and judge when feature alignment is useful.

Key terms: last N layers, activation analysis, data-driven approach, uniform, last, original, moderate compression ratios, aggressive depth pruning, feature alignment, hard or soft labels

6.6.2 Advanced techniques: when data is scarce

This section explains when advanced distillation can help with limited data and why it cannot compensate for poor data or pruning.

Key terms: Skew KLD, Feature Dynamics Distillation, compression ratio, specialized domains, quality data, solid pruning strategy, noisy, block selection, loss function

6.7 Hands-on lab

6.8 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 5: Shaping model architectures via width pruning Chapter 7: Model specialization →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.