liveBook · Manning

Depth pruning

As explained in Rearchitecting LLMs

Depth pruning is a model-reduction technique that removes complete layers from a neural network, often targeting later layers.

Depth pruning reduces a model’s size and inference cost by removing complete Transformer blocks. Each removed block discards an attention module, an MLP, and associated operations, so the smaller model may also lose capabilities. Unlike width pruning, it changes the number of sequential blocks rather than their internal dimensions. section 4.1 section 4.1.1 section 3.1.4

Removing blocks reduces parameter memory and eliminates their rounds of memory access and computation. It also removes their contribution to the key-value cache. The actual speed and capability trade-off must be measured against the original model on the intended workload. section 4.1.2 section 2.1

A practical pruning workflow:

  • Establish a baseline for capabilities, latency, and throughput before changing the model. section 2.1
  • Select candidate blocks. Position-based rules are inexpensive, but calibration data and activation measurements can reveal task-dependent importance. Compare selections experimentally rather than assuming the final blocks are always safe to remove or must always be protected. section 4.6 section 6.1
  • Remove the selected blocks, update the model configuration, and verify that it reports the actual layer count. section 2.3 section 12.5
  • Measure capability loss and inference efficiency. If needed, use knowledge distillation to recover capabilities, then compare the recovered model with the baseline. section 2.1

Remove the last two Gemma blocks

For gemma-3-270m, copy the original model so it remains available for recovery. Keep blocks 0–15 from the original 18, replace model.layers with a PyTorch ModuleList containing those blocks, and set config.num_hidden_layers to 16. Verify the resulting block list before evaluation. Removing the final blocks is a simple starting point, not necessarily the best selection. section 2.3

pruned_model.model.layers = nn.ModuleList(new_layers)
pruned_model.config.num_hidden_layers = len(pruned_model.model.layers)

The implementation replaces the block list and synchronizes the configuration with its new length. section 2.3

For data-driven selection, PyTorch forward hooks can capture block input and output activations during inference. Calibration data affects measured importance, so a selection suited to one task may not be best for another. Evaluating candidates before distillation can avoid spending substantial GPU time recovering a poorly pruned model. section 4.6 section 6.1

Measured inference trade-off

In a reported LLaMA-7B comparison, pruning 35% of the model reduced latency from 2.4 to 1.6 seconds and increased throughput from 53.7 to 80.1 tokens per second; the compared width-pruning methods slowed inference. These results illustrate an inference gain, not a guarantee for every workload or block selection. section 4.4

How Rearchitecting LLMs covers it

How deep it goes
Explains block removal, memory effects, selection strategies, PyTorch hooks, and a concrete implementation.
Examples
Prunes gemma-3-270m from 18 to 16 blocks and compares reported LLaMA-7B latency and throughput.
What you can do after
Apply it using PyTorch to remove Transformer blocks, update the configuration, and evaluate the resulting model.
How it connects
Connects Transformer blocks, width pruning, key-value caching, and knowledge distillation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.