Depth pruning reduces a model’s size and inference cost by removing complete Transformer blocks. Each removed block discards an attention module, an MLP, and associated operations, so the smaller model may also lose capabilities. Unlike width pruning, it changes the number of sequential blocks rather than their internal dimensions. section 4.1 section 4.1.1 section 3.1.4
Removing blocks reduces parameter memory and eliminates their rounds of memory access and computation. It also removes their contribution to the key-value cache. The actual speed and capability trade-off must be measured against the original model on the intended workload. section 4.1.2 section 2.1
A practical pruning workflow:
- Establish a baseline for capabilities, latency, and throughput before changing the model. section 2.1
- Select candidate blocks. Position-based rules are inexpensive, but calibration data and activation measurements can reveal task-dependent importance. Compare selections experimentally rather than assuming the final blocks are always safe to remove or must always be protected. section 4.6 section 6.1
- Remove the selected blocks, update the model configuration, and verify that it reports the actual layer count. section 2.3 section 12.5
- Measure capability loss and inference efficiency. If needed, use knowledge distillation to recover capabilities, then compare the recovered model with the baseline. section 2.1
Remove the last two Gemma blocks
For gemma-3-270m, copy the original model so it remains available for recovery. Keep blocks 0–15 from the original 18, replace model.layers with a PyTorch ModuleList containing those blocks, and set config.num_hidden_layers to 16. Verify the resulting block list before evaluation. Removing the final blocks is a simple starting point, not necessarily the best selection. section 2.3
pruned_model.model.layers = nn.ModuleList(new_layers) pruned_model.config.num_hidden_layers = len(pruned_model.model.layers)
The implementation replaces the block list and synchronizes the configuration with its new length. section 2.3
For data-driven selection, PyTorch forward hooks can capture block input and output activations during inference. Calibration data affects measured importance, so a selection suited to one task may not be best for another. Evaluating candidates before distillation can avoid spending substantial GPU time recovering a poorly pruned model. section 4.6 section 6.1
Measured inference trade-off
In a reported LLaMA-7B comparison, pruning 35% of the model reduced latency from 2.4 to 1.6 seconds and increased throughput from 53.7 to 80.1 tokens per second; the compared width-pruning methods slowed inference. These results illustrate an inference gain, not a guarantee for every workload or block selection. section 4.4
