liveBook · Manning

Chapter 4: Building smaller and faster LLMs with depth pruning

What you will learn in chapter 4 of Rearchitecting LLMs

The introduction motivates task-aware transformer block selection by questioning whether a depth-pruning heuristic that works for general-purpose models also suits specific tasks.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction motivates task-aware transformer block selection by questioning whether a depth-pruning heuristic that works for general-purpose models also suits specific tasks.

Key terms: transformer blocks, heuristic, general-purpose models, specific task

4.1 Fundamentals and benefits of depth pruning

This section defines depth pruning, states its main benefits, and clarifies the terminology used for transformer components.

Key terms: depth pruning, Transformer blocks, latency, memory consumption, knowledge recovery, block, module, layer, Attention, MLP

4.1.1 How depth pruning works

Depth pruning removes whole Transformer blocks, reducing the model’s depth, with deeper models generally better able to tolerate the loss because their blocks can be functionally redundant.

Key terms: depth pruning, block, attention, MLP, information transformation, shape, vertical dimension, depth, width, functional redundancy

4.1.2 The benefits: memory consumption and speed

This section explains how removing Transformer blocks reduces model and KV-cache memory, speeds up memory-bound inference, and makes block selection important for preserving knowledge.

Key terms: memory-bound, VRAM, depth pruning, Transformer blocks, model parameters, Qwen3-06B, KV-Cache, autoregressive generation, Key and Value tensors, attention layers

4.2 Static block selection

Introduces static block selection as a simple, effective approach that uses model structure and initial weights rather than behavior on real data.

Key terms: static approaches, model's structure, initial weights, task-specific data

4.2.1 Removing first, last, middle blocks

This section shows how to prune blocks by position, why middle-block removal can preserve useful transformations, and where a fixed position rule falls short.

Key terms: block position-based approaches, middle blocks, list of layers to keep, redundancy, fundamental representations, task-specific refinements, preservation heuristic, static rule, similar distribution, models

4.2.2 Removing by weight

This section explains how weight magnitude can rank transformer blocks for removal and why that ranking may conflict with a block-protection heuristic.

Key terms: magnitude of the weights, aggregate metric, blocks, protection heuristic, first four blocks, last two

4.2.3 Static vs. weights

This section explains why static depth-pruning rules can remove blocks that matter for a particular task, and when task-specific data is useful for choosing blocks.

Key terms: static methods, weight magnitude, architectural assumptions, block function, static approaches, task-specific data, domain-specialized optimization, data-driven methods

4.3 Data-Driven block selection

This section explains why block selection should be based on a model’s behavior on task data rather than on structure alone.

Key terms: data-driven block selection, block's importance, WinoGrande, coreference resolution, PyTorch hooks, input, output, passive, cosine similarity

4.3.1 Using PyTorch hooks

This section shows how to use PyTorch forward hooks to inspect transformer-block inputs and outputs without changing the model’s code.

Key terms: hook, module, input, output, forward method, activations, model->layers, register_forward_hook, handle, hook_handle.remove()

4.3.2 Understanding cosine similarity

Explains why cosine similarity measures changes in vector direction and how to calculate and interpret it for a transformer block’s input and output activations.

Key terms: cosine similarity, Euclidean distance, magnitude, direction, input vector, output vector, block, normalize, dot product, torch.mm

4.3.3 Analyzing block contributions with WinoGrande

This section builds a WinoGrande-calibrated pipeline that captures each transformer block’s activations and ranks blocks by how much they change their inputs.

Key terms: WinoGrande, coreference resolution, calibration, setup_layer_hooks, create_hook, layer_idx, .detach(), activations, calculate_cosine_importance, cosine similarity

4.3.4 Choosing the blocks to discard

This section uses block-importance scores and optional protection rules to choose blocks for depth pruning, while showing why the choice depends on calibration data and cannot be settled by scores alone.

Key terms: importance scores, block, static magnitude, data-driven cosine similarity, Winogrande, block protection heuristic, calibration dataset, WikiText, removal candidate, protection heuristic

4.3.5 Analysis of the benchmarks

This section compares how block-selection strategies affect benchmark quality and inference speed after depth pruning.

Key terms: WinoGrande, Lambada, final blocks, chance, data guided protected, adjacent protection, mid blocks, long-range dependency, cosine signal, data guided unprotected

4.4 From paper to practice

This section connects the chapter’s depth-pruning choices to published results on inference speed, block selection, and performance recovery.

Key terms: depth pruning, width pruning, latency, throughput, Perplexity, Taylor approximation, calibration dataset, gradients, forward inference pass, ShortGPT

4.5 Hands-on lab

4.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 3: A blueprint to modern transformers Chapter 5: Shaping model architectures via width pruning →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.