Chapter 4: Building smaller and faster LLMs with depth pruning
The introduction motivates task-aware transformer block selection by questioning whether a depth-pruning heuristic that works for general-purpose models also suits specific tasks.
After this chapter you will be able to
- Task-aware block selection rationale. Explain why leaving the first and last transformer blocks untouched may not be enough when pruning a model for a specific task. The heuristic has worked well for general-purpose models, but the blocks to remove may depend on the data the model will process.
- Depth pruning and benefits. Explain that depth pruning removes entire Transformer blocks and, when done well, can reduce latency, memory consumption, and the cost of subsequent knowledge recovery.
- Whole block removal. Explain that depth pruning removes an entire Transformer block, not just one of its modules, and therefore discards a complete information transformation learned during training.
- Model depth and redundancy. Describe how removing blocks changes a model’s shape by reducing its number of sequential transformations, and explain why models that start with more blocks generally tolerate this better.
- Memory Bound Inference Speed. Explain why depth pruning can speed up LLM inference primarily by reducing data movement rather than just computation.
- KV Cache Growth Reduction. Explain why removing attention-containing blocks reduces KV-cache memory growth during token-by-token generation.
- Efficiency Versus Knowledge Loss. Distinguish what determines the efficiency gain from what determines knowledge loss when selecting blocks to prune.
- Static selection basis and fit. Explain what static block selection uses instead of real-data behavior, and why it can be a good choice for general-purpose models or when task-specific data is unavailable.
- Middle block removal procedure. Describe how to remove middle blocks by identifying the blocks to discard, joining the lists of blocks before and after them, and assigning the retained blocks to the new model.
- Position preservation rationale. Explain why the section favors removing potentially redundant middle blocks while preserving early and late blocks, and describe the stated preservation heuristic.
- Weight magnitude block ranking. Explain how the method uses an aggregate weight-magnitude score for each block to select blocks for removal, and state the assumption behind that choice.
- Limits of static block selection. Explain why weight magnitude and architectural rules do not reliably indicate which blocks are safe to remove for a particular task.
- Task-relative block importance. Explain why a block’s importance depends on the data and task rather than being an absolute property of the model. Describe WinoGrande as the task data proposed for identifying blocks that contribute least to coreference resolution.
- Passive block identification. Describe how observing each block’s input and output can reveal blocks that barely change the representation. Explain why those blocks are candidates for removal in the specific data or task context.
- Forward hook inputs and outputs. Explain when a forward hook runs, what its module, input, and output arguments contain, and why those arguments let you compare a block’s input and output activations.
- Hook registration and cleanup. Describe how the example attaches a forward hook to a transformer block, triggers it with a model query, and removes it when inspection is finished.
- Direction versus vector magnitude. Explain why cosine similarity is used instead of Euclidean distance to compare activation vectors: vector magnitude can change without meaning changing, while direction carries semantic information.
- Similarity scores and block changes. Describe what cosine similarity values of 1, 0, and -1 mean for vector directions, and explain how similarity between a block’s input and output indicates how much the block transforms the information.
- Per-block activation hooks. Explain how hooks capture activations separately for each block and why the hook factory retains a different layer index for each one. Describe why captured tensors are detached.
- Cosine-based block importance. Explain how average input-to-output cosine similarity becomes a block importance score and why padding tokens are excluded from the average.
- Dataset-wide block ranking. Trace how the pipeline registers hooks, runs WinoGrande data through the model, calculates block scores, and aggregates them across batches into a ranking.
- Limits of importance scores. Explain why a low block-importance score is a guide to pruning rather than proof that removing the block will preserve performance.
- Calibration-dependent removal candidates. Explain how changing the calibration dataset affects which blocks become removal candidates in this example.
- Protected versus unprotected selection. Describe the two optional safeguards and trace how using them changes the four-block selection in this example.
- Benchmark sensitivity to final blocks. Explain why WinoGrande and Lambada show different degrees of damage after pruning, and describe how removing or preserving final blocks affects the reported results.
- Protected versus scattered removals. Describe the contrasting results of protecting final blocks and scattering removals across middle blocks.
- Inference gains from block count. Describe the inference metrics and results, including why the two pruned models had similar speed despite removing different blocks.
- Depth pruning speed advantage. Explain why removing entire blocks can accelerate small-batch inference more effectively than width pruning, and cite the reported LLaMA-7B result as evidence.
- Block importance metric trade-offs. Distinguish perplexity testing, Taylor approximation, and the chapter’s forward-pass approach by what each measures and the work it requires.
- Block Influence research evidence. Describe ShortGPT’s Block Influence metric and explain what its reported pruning results and importance correlation support.
What it builds on
- Applying depth pruning (chapter 2) · Assumes familiarity with removing transformer blocks through depth pruning.
- The Transformer dimensions: depth and width (chapter 3) · Introduces transformer depth as an architectural dimension.
- General behaviour of a Transformer model (chapter 3) · The general behavior of a Transformer model.
Section by section
Introduction
The introduction motivates task-aware transformer block selection by questioning whether a depth-pruning heuristic that works for general-purpose models also suits specific tasks.
- Task-aware block selection rationale: Explain why leaving the first and last transformer blocks untouched may not be enough when pruning a model for a specific task. The heuristic has worked well for general-purpose models, but the blocks to remove may depend on the data the model will process.
Why it matters: It explains why block selection deserves reconsideration when the pruned model has a particular task.
Key terms: transformer blocks, heuristic, general-purpose models, specific task
4.1 Fundamentals and benefits of depth pruning
This section defines depth pruning, states its main benefits, and clarifies the terminology used for transformer components.
- Depth pruning and benefits: Explain that depth pruning removes entire Transformer blocks and, when done well, can reduce latency, memory consumption, and the cost of subsequent knowledge recovery.
Why it matters: This connects the structural change to the benefits the chapter aims to achieve. - Block module layer terminology: Distinguish a Transformer block from a module and a layer using the section’s examples, and state where HuggingFace stores blocks in the cited model code.
Why it matters: These distinctions make it clear what is—and is not—removed by depth pruning.
Key terms: depth pruning, Transformer blocks, latency, memory consumption, knowledge recovery, block, module, layer, Attention, MLP
4.1.1 How depth pruning works
Depth pruning removes whole Transformer blocks, reducing the model’s depth, with deeper models generally better able to tolerate the loss because their blocks can be functionally redundant.
- Whole block removal: Explain that depth pruning removes an entire Transformer block, not just one of its modules, and therefore discards a complete information transformation learned during training.
Why it matters: Removing a block is more consequential than removing an individual component within it. - Model depth and redundancy: Describe how removing blocks changes a model’s shape by reducing its number of sequential transformations, and explain why models that start with more blocks generally tolerate this better.
Why it matters: The model’s starting depth helps explain why the same kind of removal may have different effects on different models.
Key terms: depth pruning, block, attention, MLP, information transformation, shape, vertical dimension, depth, width, functional redundancy
4.1.2 The benefits: memory consumption and speed
This section explains how removing Transformer blocks reduces model and KV-cache memory, speeds up memory-bound inference, and makes block selection important for preserving knowledge.
- Memory Bound Inference Speed: Explain why depth pruning can speed up LLM inference primarily by reducing data movement rather than just computation.
Why it matters: It explains why removing blocks can improve speed even when GPU arithmetic is not fully utilized. - Model Weight Memory Savings: Describe how removing blocks reduces the memory needed to load a model, using the section's Qwen3-06B example.
Why it matters: Block removal saves memory for model weights as well as reducing memory traffic. - KV Cache Growth Reduction: Explain why removing attention-containing blocks reduces KV-cache memory growth during token-by-token generation.
Why it matters: KV-cache savings become increasingly important as generated sequences grow. - Efficiency Versus Knowledge Loss: Distinguish what determines the efficiency gain from what determines knowledge loss when selecting blocks to prune.
Why it matters: It explains why equal-sized pruned models can have different quality and recovery costs.
Key terms: memory-bound, VRAM, depth pruning, Transformer blocks, model parameters, Qwen3-06B, KV-Cache, autoregressive generation, Key and Value tensors, attention layers
4.2 Static block selection
Introduces static block selection as a simple, effective approach that uses model structure and initial weights rather than behavior on real data.
- Static selection basis and fit: Explain what static block selection uses instead of real-data behavior, and why it can be a good choice for general-purpose models or when task-specific data is unavailable.
Why it matters: This gives you a basis for deciding when static selection is appropriate without relying on task-specific examples.
Key terms: static approaches, model's structure, initial weights, task-specific data
4.2.1 Removing first, last, middle blocks
This section shows how to prune blocks by position, why middle-block removal can preserve useful transformations, and where a fixed position rule falls short.
- Middle block removal procedure: Describe how to remove middle blocks by identifying the blocks to discard, joining the lists of blocks before and after them, and assigning the retained blocks to the new model.
Why it matters: The retained-block list determines which transformations remain in the pruned model. - Position preservation rationale: Explain why the section favors removing potentially redundant middle blocks while preserving early and late blocks, and describe the stated preservation heuristic.
Why it matters: The rationale explains what middle-block pruning tries to retain, not just which blocks it removes. - Static selection limitation: Explain why preserving blocks by position remains a static rule despite being more robust than simply removing blocks from one end.
Why it matters: A useful positional heuristic is not a guarantee that the same blocks should be kept for every model or task.
Key terms: block position-based approaches, middle blocks, list of layers to keep, redundancy, fundamental representations, task-specific refinements, preservation heuristic, static rule, similar distribution, models
4.2.2 Removing by weight
This section explains how weight magnitude can rank transformer blocks for removal and why that ranking may conflict with a block-protection heuristic.
- Weight magnitude block ranking: Explain how the method uses an aggregate weight-magnitude score for each block to select blocks for removal, and state the assumption behind that choice.
Why it matters: The ranking only makes sense if you understand the assumption connecting weight size to block contribution. - Protection heuristic conflict: Describe how the weight-based selection can conflict with the protection heuristic and whether that heuristic must be applied.
Why it matters: A weight-based removal list does not by itself settle whether protected blocks should be removed.
Key terms: magnitude of the weights, aggregate metric, blocks, protection heuristic, first four blocks, last two
4.2.3 Static vs. weights
This section explains why static depth-pruning rules can remove blocks that matter for a particular task, and when task-specific data is useful for choosing blocks.
- Limits of static block selection: Explain why weight magnitude and architectural rules do not reliably indicate which blocks are safe to remove for a particular task.
Why it matters: A simple pruning rule can remove exactly the capability you need to keep. - Choosing task-aware block selection: Distinguish when static approaches are useful from when data-driven block selection is preferable.
Why it matters: The data available and the intended use case should shape how you choose blocks to prune.
Key terms: static methods, weight magnitude, architectural assumptions, block function, static approaches, task-specific data, domain-specialized optimization, data-driven methods
4.3 Data-Driven block selection
This section explains why block selection should be based on a model’s behavior on task data rather than on structure alone.
- Task-relative block importance: Explain why a block’s importance depends on the data and task rather than being an absolute property of the model. Describe WinoGrande as the task data proposed for identifying blocks that contribute least to coreference resolution.
Why it matters: It explains why the same block may be a different pruning candidate for a different task. - Passive block identification: Describe how observing each block’s input and output can reveal blocks that barely change the representation. Explain why those blocks are candidates for removal in the specific data or task context.
Why it matters: It connects observations of a running model to a reason for considering particular blocks for pruning.
Key terms: data-driven block selection, block's importance, WinoGrande, coreference resolution, PyTorch hooks, input, output, passive, cosine similarity
4.3.1 Using PyTorch hooks
This section shows how to use PyTorch forward hooks to inspect transformer-block inputs and outputs without changing the model’s code.
- Forward hook inputs and outputs: Explain when a forward hook runs, what its module, input, and output arguments contain, and why those arguments let you compare a block’s input and output activations.
Why it matters: Seeing both sides of a block’s computation makes it possible to examine how much that block changes its activations. - Hook registration and cleanup: Describe how the example attaches a forward hook to a transformer block, triggers it with a model query, and removes it when inspection is finished.
Why it matters: Registering and removing the hook lets you inspect a live model without leaving the inspection function attached. - Printed tensor shape interpretation: Explain what the example’s printed module class and tensor shape reveal about the hooked block and its input.
Why it matters: The printed output checks both that the intended block was hooked and how to read the activations passing through it.
Key terms: hook, module, input, output, forward method, activations, model->layers, register_forward_hook, handle, hook_handle.remove()
4.3.2 Understanding cosine similarity
Explains why cosine similarity measures changes in vector direction and how to calculate and interpret it for a transformer block’s input and output activations.
- Direction versus vector magnitude: Explain why cosine similarity is used instead of Euclidean distance to compare activation vectors: vector magnitude can change without meaning changing, while direction carries semantic information.
Why it matters: This explains why the comparison focuses on a block’s change in direction rather than simply the distance between its activations. - Similarity scores and block changes: Describe what cosine similarity values of 1, 0, and -1 mean for vector directions, and explain how similarity between a block’s input and output indicates how much the block transforms the information.
Why it matters: The score gives a way to interpret how much a block changed its input activations. - Normalized dot product calculation: Describe the two calculation steps for cosine similarity—normalize the vectors, then take their dot product—and state how the dot product produces a single score.
Why it matters: This connects the directional interpretation of cosine similarity to the calculation used in the example.
Key terms: cosine similarity, Euclidean distance, magnitude, direction, input vector, output vector, block, normalize, dot product, torch.mm
4.3.3 Analyzing block contributions with WinoGrande
This section builds a WinoGrande-calibrated pipeline that captures each transformer block’s activations and ranks blocks by how much they change their inputs.
- WinoGrande calibration text: Describe how WinoGrande examples are formatted for the block analysis: each text includes a sentence with a blank, two candidate entities, and the correct answer written into the text.
Why it matters: The block scores depend on the examples passed through the model. - Per-block activation hooks: Explain how hooks capture activations separately for each block and why the hook factory retains a different layer index for each one. Describe why captured tensors are detached.
Why it matters: Separate, detached captures make it possible to compare each block’s input and output without retaining gradient history. - Cosine-based block importance: Explain how average input-to-output cosine similarity becomes a block importance score and why padding tokens are excluded from the average.
Why it matters: This score gives the pipeline a consistent way to rank blocks by their measured transformations on the calibration text. - Dataset-wide block ranking: Trace how the pipeline registers hooks, runs WinoGrande data through the model, calculates block scores, and aggregates them across batches into a ranking.
Why it matters: The aggregated ranking is the output needed before deciding which blocks to remove.
Key terms: WinoGrande, coreference resolution, calibration, setup_layer_hooks, create_hook, layer_idx, .detach(), activations, calculate_cosine_importance, cosine similarity
4.3.4 Choosing the blocks to discard
This section uses block-importance scores and optional protection rules to choose blocks for depth pruning, while showing why the choice depends on calibration data and cannot be settled by scores alone.
- Limits of importance scores: Explain why a low block-importance score is a guide to pruning rather than proof that removing the block will preserve performance.
Why it matters: A ranking helps narrow the choices, but it cannot guarantee the effect of pruning on performance. - Contrasting block importance rankings: Describe how static magnitude and Winogrande-calibrated cosine similarity give different views of block importance, especially near the end of the model.
Why it matters: Different measures can point to different pruning candidates even in the same model. - Calibration-dependent removal candidates: Explain how changing the calibration dataset affects which blocks become removal candidates in this example.
Why it matters: The data used to rank blocks can change the architecture selected for pruning. - Protected versus unprotected selection: Describe the two optional safeguards and trace how using them changes the four-block selection in this example.
Why it matters: Safeguards change which blocks are removed without changing the number removed, making the strategies possible to compare.
Key terms: importance scores, block, static magnitude, data-driven cosine similarity, Winogrande, block protection heuristic, calibration dataset, WikiText, removal candidate, protection heuristic
4.3.5 Analysis of the benchmarks
This section compares how block-selection strategies affect benchmark quality and inference speed after depth pruning.
- Benchmark sensitivity to final blocks: Explain why WinoGrande and Lambada show different degrees of damage after pruning, and describe how removing or preserving final blocks affects the reported results.
Why it matters: A benchmark can make a damaging pruning choice look less harmful than it is. - Protected versus scattered removals: Describe the contrasting results of protecting final blocks and scattering removals across middle blocks.
Why it matters: Where removed blocks sit matters for quality, and a protection heuristic is not automatically helpful in every selection strategy. - Dataset-dependent cosine selections: Explain what the WinoGrande and WikiText block selections show about data-guided pruning in this experiment.
Why it matters: A data-guided score is not a fixed ranking independent of the dataset used to calculate it. - Inference gains from block count: Describe the inference metrics and results, including why the two pruned models had similar speed despite removing different blocks.
Why it matters: Block placement strongly affects retained quality, while the measured speed benefit mainly follows the number of blocks removed.
Key terms: WinoGrande, Lambada, final blocks, chance, data guided protected, adjacent protection, mid blocks, long-range dependency, cosine signal, data guided unprotected
4.4 From paper to practice
This section connects the chapter’s depth-pruning choices to published results on inference speed, block selection, and performance recovery.
- Depth pruning speed advantage: Explain why removing entire blocks can accelerate small-batch inference more effectively than width pruning, and cite the reported LLaMA-7B result as evidence.
Why it matters: It explains why depth pruning is attractive when inference speed is a primary goal. - Block importance metric trade-offs: Distinguish perplexity testing, Taylor approximation, and the chapter’s forward-pass approach by what each measures and the work it requires.
Why it matters: The comparison explains the practical cost behind the chapter’s choice of block-selection metric. - Block Influence research evidence: Describe ShortGPT’s Block Influence metric and explain what its reported pruning results and importance correlation support.
Why it matters: It shows why the chapter’s cosine-based block selection is more than an ad-hoc shortcut. - Boundary block protection approaches: Distinguish how the two papers protect important blocks near the beginning and end of a model.
Why it matters: It distinguishes a fixed protection rule from protection that emerges from importance scoring. - Post-pruning recovery choices: Describe the recovery approaches reported by the two papers and explain the book’s stated distinction between general-capability recovery and domain specialization.
Why it matters: It places the book’s planned recovery choice alongside alternatives without treating pruning alone as the finished model.
Key terms: depth pruning, width pruning, latency, throughput, Perplexity, Taylor approximation, calibration dataset, gradients, forward inference pass, ShortGPT
4.5 Hands-on lab
4.6 Summary
Where this is used later
- (chapter intro) (chapter 5) · Depth pruning as removal of Transformer blocks.
- Reconstructing the MLP module (chapter 5) · Depth pruning removes Transformer blocks.
- (chapter intro) (chapter 6) · Depth pruning removes Transformer blocks.
- What the KV cache is and why it matters (chapter 8) · Revisits what the KV cache is and why its memory footprint matters.
- Selecting the best candidate (chapter 6) · Supports selecting a pruning candidate before knowledge recovery.
- Choosing what to prune (chapter 6) · Static block selection was introduced earlier.
- Selecting static neurons (chapter 5) · Selecting pruning candidates using weight magnitude.
- Extracting the dynamic importance component (chapter 5) · Introduces PyTorch hooks for capturing activations during forward passes.
Concepts explained in this chapter
- cosine similarityCosine similarity measures the cosine of the angle between two vectors, reflecting their directional alignment independently of magnitude.
- protection heuristicA protection heuristic is a rule that excludes selected model components from pruning to reduce the risk of damaging model capabilities.
- hookA model hook is a callback attached to a neural-network component that observes or modifies computation at a specified point during execution.
- depth pruningDepth pruning is a model-reduction technique that removes complete layers from a neural network, often targeting later layers.
- feature importance scoreAn importance score numerically estimates how important a model component is to its information or performance.
- data-driven methodData-driven pruning selects model components for removal using measurements from representative input data.
- adjacent block protectionAdjacent block protection is a pruning constraint that prevents or limits the removal of neighboring model blocks.
- static pruningStatic pruning selects model components for removal using fixed structural information or parameter statistics rather than runtime activation behavior.
- cosine importanceCosine importance estimates a model component's importance using cosine similarity between related activation or representation vectors.
- throughputThroughput is the amount of model work or output that a system completes per unit of time.
- Taylor approximationA Taylor approximation estimates the change in a model's loss caused by removing a component using a local approximation based on derivatives.
- block influenceBlock Influence is a metric defined as one minus the cosine similarity between a model block's input and output representations.
- continued pretrainingContinued pretraining is additional unsupervised training of a pretrained model on text or other pretraining data.
- perplexityPerplexity is a language-modeling metric that measures how well a model predicts a sequence, with lower values indicating better predictive performance.
- long-range contextLong-range context is information occurring substantially earlier in a sequence that remains relevant to predicting a later token.
- time to first tokenTime to first token is the elapsed time from a generation request until the model produces its first output token.
- coreference resolutionCoreference resolution is the task of determining which previously mentioned entity a referring expression, such as a pronoun, denotes.
- transformer blockA transformer layer, or transformer block, is a processing unit that uses attention and feed-forward components to transform representations.
- memory boundMemory-bound describes a workload whose performance is limited primarily by memory transfers rather than arithmetic processing.
- prune_modelprune_model is a function that removes selected structural components from a neural model to produce a pruned model.
- register_forward_hookregister_forward_hook is a PyTorch method that registers a callback to run after a module's forward computation.
- block selection strategyA block selection strategy is a method for choosing which transformer blocks to retain or remove during depth pruning.
- capability retentionCapability retention is the proportion of a model's evaluated performance preserved after a structural modification such as pruning.
- magnitude-based selectionMagnitude-based selection chooses model components for pruning according to the magnitudes of their parameters or related scores.
- inference performanceInference speed is the rate at which a trained model produces outputs from inputs.
- key-value cacheA key-value cache stores previously computed attention key and value tensors to accelerate autoregressive inference.
- memory usageMemory consumption is the amount of computer memory required to store and execute a model or computation.
- data-driven selectionData-driven selection is the choice of model components based on their measured behavior on task-relevant data.
- calibration dataCalibration data is representative input data used to measure model behavior and guide data-dependent model compression decisions.
- autoregressive generationAutoregressive generation produces a sequence one token at a time, conditioning each new token on previously generated tokens.
- neural network layerA model layer is a computational stage in a neural network that transforms incoming representations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
