Width pruning reduces a model’s internal width by removing selected neurons or channels. In a Transformer MLP, it narrows the intermediate expansion while preserving the hidden size at block boundaries. Unlike depth pruning, it leaves the Transformer blocks and attention layers intact. section 3.1.4 section 5.5
A GLU MLP expands its input through parallel gate_proj and up_proj layers, combines their outputs element-wise, and contracts the result through down_proj. Pruning must remove corresponding positions from both expansion projections and the matching inputs to the contraction projection. section 5.1 section 6.5.3
Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.
The GLU diagram shows the intermediate dimension being narrowed while the input and output dimensions remain unchanged. section 5.1
A practical pruning workflow:
- Score each neuron pair using gate_proj and up_proj weights. One static metric sums max(weight) + |min(weight)| for the two projections; low-scoring pairs are removal candidates. section 5.3 section 5.1
- For data-driven selection, collect L2 activation norms at down_proj’s input with PyTorch hooks, then multiply those norms by the structural scores. Check that calibration data exists for every layer. section 5.5 section 5.2.4
- Keep the highest-scoring positions, rebuild the three projections with matching indices, and update intermediate_size in the model configuration. Copy the original model first so it remains available as a baseline. section 5.1.2 section 5.2.4
- Test that generation works, then compare capabilities and perplexity against the baseline. Evaluate several tasks: different capabilities can degrade at different rates. section 5.1.3
Figure 5.8 Model surgery using hybrid importance. The update_model function (Listing 5.7) iterates over each layer (1-2), calls prune_neuron_pairs (3) that calculates hybrid scores by multiplying structural magnitudes by activation norms (4), selects the top-K neurons and creates new layers (5), replaces the original layers (6), and returns the specialized model (7).
This diagram traces calibration scores through neuron selection, layer reconstruction, and model replacement. section 5.2.4
Static pruning and capability trade-offs
Removing 40% of the MLP expansion neurons from Llama-3.2-1B reduced parameters by about 322 million, or 26.06%, while keeping all 16 blocks. Without recovery training, WikiText perplexity rose from 11.57 to 56.33 and GSM8K fell from 0.0660 to 0.0205. Smaller size did not preserve all capabilities equally. section 5.1.2 section 5.1.3 section 5.3
Calibration data changes which neurons survive: WikiText and SMS calibration can produce models with the same width but different retained connections. Width pruning can also precede depth pruning and knowledge distillation to recover lost knowledge. section 5.2.4 section 6.5.3
For deployment, benchmark on the target hardware. An unaligned intermediate size can cause matrix kernels to pad the width, while different widths across layers may not fit serving-engine assumptions. Alignment to hardware-friendly multiples is a useful experiment, not a guarantee of speedup. section 12.5 section 5.4
