Chapter 2: An end-to-end rearchitecting project
Introduces a general-purpose project to make a base model smaller and faster while preserving its capabilities through structural optimization and knowledge recovery.
After this chapter you will be able to
- Model adaptation objective. Explain that the project aims to reduce the size of a working base model while preserving as many of its capabilities as possible, rather than optimizing it for a specific domain or task.
- Complementary rearchitecting phases. Distinguish structural optimization, which changes the architecture to make the model lighter and faster, from knowledge recovery, which transfers knowledge from the base model to the pruned model.
- Pruning and recovery workflow. Trace the project from measuring an original model through depth pruning, evaluating lost capabilities, knowledge recovery, and comparison with the original.
- Two dimensions of performance. Explain the section’s definition of model performance in terms of capabilities and inference efficiency.
- Quantified comparison baseline. Explain why the base model needs a measured baseline and how its benchmark scores will be used to judge later models.
- Pruned model reconstruction. Describe how the example reduces the copied model from 18 layers to 16 and keeps PyTorch registration and the model configuration consistent with that change.
- Coherence versus output quality. Distinguish the pruned model's ability to produce coherent text from its factual accuracy and focus, using the Paris response as evidence.
- Pruning quality speed tradeoff. Describe the measured changes after removing two layers without adaptation: a smaller, faster model with lower benchmark performance.
- Teacher and student roles. Describe which models serve as teacher and student, why the student needs recovery, and how the student is created from a copy of the teacher.
- Prediction distribution matching. Explain what the student imitates in logits-only distillation and how KL divergence guides that imitation.
- Distillation training cycle. Trace one batch through teacher and student predictions, loss calculation, and student weight updates; distinguish why gradients are disabled for the teacher but retained for the student.
- Final benchmark trade-off. Explain how the recovered student model performed when evaluated against the original model after knowledge distillation.
What it builds on
- The solution: the model rearchitecting pipeline (chapter 1) · The rearchitecting pipeline introduced in the previous chapter.
- How data moves through the pipeline (chapter 1) · The earlier overview of how data moves through the rearchitecting pipeline.
Section by section
Introduction
Introduces a general-purpose project to make a base model smaller and faster while preserving its capabilities through structural optimization and knowledge recovery.
- Model adaptation objective: Explain that the project aims to reduce the size of a working base model while preserving as many of its capabilities as possible, rather than optimizing it for a specific domain or task.
Why it matters: This objective explains why the chapter considers both model size and retained capability. - Complementary rearchitecting phases: Distinguish structural optimization, which changes the architecture to make the model lighter and faster, from knowledge recovery, which transfers knowledge from the base model to the pruned model.
Why it matters: The two phases address different parts of the smaller-model goal: efficiency and retained capability.
Key terms: base model, general-purpose optimization, capabilities, Structural Optimization, Knowledge Recovery, pruned model
2.1 The rearchitecting workflow
This section outlines a practical pruning-and-recovery workflow and defines how the project will judge the resulting model.
- Pruning and recovery workflow: Trace the project from measuring an original model through depth pruning, evaluating lost capabilities, knowledge recovery, and comparison with the original.
Why it matters: The sequence explains why each measurement and intervention appears in the project. - Two dimensions of performance: Explain the section’s definition of model performance in terms of capabilities and inference efficiency.
Why it matters: A smaller model is only a useful result if its efficiency gains are considered alongside its capabilities. - Project model hardware fit: Justify the choice of gemma-3-270m for this first hands-on project.
Why it matters: The model choice makes the practical project feasible on accessible hardware.
Key terms: baseline, depth pruning, knowledge distillation, Structural Optimization, Knowledge Recovery, model performance, capabilities, inference efficiency, tokens per second, latency
2.2 Establishing the baseline
This section establishes the base model’s architecture, generation, benchmark-score, and inference-time baselines so later changes can be compared against them.
- Quantified comparison baseline: Explain why the base model needs a measured baseline and how its benchmark scores will be used to judge later models.
Why it matters: Without a measured starting point, later pruning and recovery results have no clear comparison. - Base model size and structure: Describe why Gemma 3 270M is practical for this project and identify the structural facts inspected before modifying it.
Why it matters: Knowing the starting structure makes subsequent changes to the model interpretable. - Generation sanity check: Explain what the simple generation test checks and why a coherent answer does not establish complete factual accuracy.
Why it matters: A quick, repeatable response checks that generation works without mistaking fluent output for reliable facts. - Benchmark capability coverage: Distinguish what each of the four selected benchmarks tests, including how LAMBADA differs from the multiple-choice tasks.
Why it matters: The scores are more informative when you can say which kinds of behavior they represent. - Inference-time baseline: Describe how response time is measured as a simple efficiency baseline and why measured times may vary.
Why it matters: Timing supplies an efficiency comparison, but small timing differences need to be interpreted cautiously.
Key terms: performance baseline, lm-eval, model_evaluation, google/gemma-3-270m, parameters, Gemma3DecoderLayer, layers, sanity check, generate_text, do_sample=False
2.3 Applying depth pruning
The section removes the final two transformer blocks from a copy of a model, updates its structure and configuration, and explains the resulting parameter reduction.
- Example layer selection: Explain why this example removes the last two transformer blocks and why that choice is not presented as optimal.
Why it matters: It keeps a convenient demonstration choice from being mistaken for a proven pruning strategy. - Original model preservation: Explain why the pruning modifications are made to a separate copy of the model.
Why it matters: Keeping both versions available prepares the model for the recovery step. - Pruned model reconstruction: Describe how the example reduces the copied model from 18 layers to 16 and keeps PyTorch registration and the model configuration consistent with that change.
Why it matters: Removing blocks is not enough by itself: the retained layers must remain registered and the configuration must describe the model that actually exists. - Layer and parameter reductions: Explain why removing two of the 18 layers reduces the model's parameter count by only 4.16%, rather than by the same proportion as its layer count.
Why it matters: Layer-count reduction alone does not tell you how much smaller the model's parameter footprint becomes.
Key terms: depth pruning, transformer blocks, final layers, original model, modified version, knowledge recovery, model.layers, nn.ModuleList, num_hidden_layers, parameters
2.4 Evaluating the impact of pruning
The section compares a model pruned by two layers with its baseline, showing that faster inference comes with degraded output and benchmark performance.
- Coherence versus output quality: Distinguish the pruned model's ability to produce coherent text from its factual accuracy and focus, using the Paris response as evidence.
Why it matters: A fluent sample alone can hide damage caused by pruning. - Pruning quality speed tradeoff: Describe the measured changes after removing two layers without adaptation: a smaller, faster model with lower benchmark performance.
Why it matters: The comparison makes the cost of the speed gain visible before any recovery work. - Winogrande score caveat: Explain why the slight increase in the Winogrande score does not demonstrate preserved reasoning.
Why it matters: A small score increase can be misleading when both scores are close to guessing.
Key terms: coherent text, hallucinations, degradation, baseline, benchmarks, inference time, depth pruning, Winogrande, random chance
2.5 Recovering knowledge
This section shows how to prepare data and use logits-only knowledge distillation to help a depth-pruned model recover lost capabilities.
- Teacher and student roles: Describe which models serve as teacher and student, why the student needs recovery, and how the student is created from a copy of the teacher.
Why it matters: The teacher–student distinction explains which model provides the target and which model changes during recovery. - Recovery data preparation: Trace how the section turns a subset of general-purpose text into batches for distillation, and explain why that dataset suits the recovery goal.
Why it matters: Distillation needs varied example text converted into batches that both models can process. - Prediction distribution matching: Explain what the student imitates in logits-only distillation and how KL divergence guides that imitation.
Why it matters: Matching the full distribution gives the student a richer training target than the teacher’s final answer alone. - Distillation training cycle: Trace one batch through teacher and student predictions, loss calculation, and student weight updates; distinguish why gradients are disabled for the teacher but retained for the student.
Why it matters: The cycle shows how the teacher’s predictions become weight changes in the student without retraining the teacher.
Key terms: Teacher model, Student model, depth pruning, prune_model, SlimPajama, tokenization, input_ids, attention_mask, Dataset, DataLoader
2.6 Analyzing the final result
This section assesses the first rearchitecture cycle through repeated benchmarks and a qualitative check of the recovered model’s answer about Paris.
- Final benchmark trade-off: Explain how the recovered student model performed when evaluated against the original model after knowledge distillation.
Why it matters: These results show the size-and-speed gains alongside the remaining capability gap. - Paris response recovery: Describe what the final Paris test showed about the recovered model’s response.
Why it matters: The Paris example gives a concrete check of the recovery beyond the controlled benchmarks.
Key terms: student model, knowledge distillation, controlled benchmarks, pruned model, recovered model, base model
2.7 Hands-on lab
2.8 Summary
Where this is used later
- Removing first, last, middle blocks (chapter 4) · The reader compares removing blocks from different positions.
- The Transformer dimensions: depth and width (chapter 3) · Introduced depth pruning by removing blocks.
- (chapter intro) (chapter 4) · Assumes familiarity with removing transformer blocks through depth pruning.
- Aligning Teacher and Student (chapter 6) · Teacher and student roles underpin their alignment for distillation.
- (chapter intro) (chapter 6) · Recovering knowledge after pruning.
- Choosing what to prune (chapter 6) · The earlier rearchitecting workflow introduces knowledge recovery after pruning.
- Preparing the recovery dataset (chapter 6) · Recovery dataset preparation is developed further.
- The compound loss function (chapter 6) · The loss-function discussion builds on comparing teacher and student predictions.
Concepts explained in this chapter
- knowledge recoveryKnowledge recovery is a model-rearchitecting phase that restores or preserves model capabilities after structural changes.
- distillationKnowledge distillation is a model-compression technique in which a smaller student model learns to reproduce useful behavior from a larger teacher model.
- benchmarkA benchmark is a standardized task or test suite used to measure and compare model performance.
- inference performanceInference speed is the rate at which a trained model produces outputs from inputs.
- foundation modelA foundation model is a broadly pretrained model intended to support adaptation to many downstream tasks.
- baselineA quantitative baseline is an objective reference measurement established before interventions to evaluate their effects.
- student modelA student model is a model trained to reproduce a teacher model's behavior during knowledge distillation.
- transformer blockA transformer layer, or transformer block, is a processing unit that uses attention and feed-forward components to transform representations.
- LAMBADALAMBADA is a language-modeling benchmark that evaluates whether a model can predict a target word from a textual passage using its preceding context.
- parameter countParameter count is the number of learned scalar parameters in a machine-learning model.
- teacher modelA teacher model is a model whose outputs or behavior guide the training of a student model during knowledge distillation.
- Kullback–Leibler divergenceKullback–Leibler divergence measures how one probability distribution differs from a reference distribution.
- logitA logit is an unnormalized numerical score produced by a model before conversion into probabilities.
- tokenizationTokenization is the process of converting text into discrete numerical units that a language model can process.
- training loopA training loop is an iterative procedure that computes model outputs, evaluates an error, and updates trainable parameters over batches and iterations.
- neural network layerA model layer is a computational stage in a neural network that transforms incoming representations.
- depth pruningDepth pruning is a model-reduction technique that removes complete layers from a neural network, often targeting later layers.
- pruned modelA pruned model is a model from which selected parameters or structural components have been removed.
- pruningA pruning strategy is a method for deciding which model components to remove during model compression.
- structural optimizationStructural optimization improves a model by changing its architecture, components, or organization.
- structured pruningStructured pruning is a model-compression technique that removes organized groups of parameters, such as neurons, channels, or layers.
- model rearchitectingModel rearchitecting is the deliberate modification of a model's architecture or components to meet particular requirements or constraints.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
