liveBook · Manning

Width pruning

As explained in Rearchitecting LLMs

Width pruning is a structured pruning technique that removes selected neurons or channels to reduce a model's width.

Width pruning reduces a model’s internal width by removing selected neurons or channels. In a Transformer MLP, it narrows the intermediate expansion while preserving the hidden size at block boundaries. Unlike depth pruning, it leaves the Transformer blocks and attention layers intact. section 3.1.4 section 5.5

A GLU MLP expands its input through parallel gate_proj and up_proj layers, combines their outputs element-wise, and contracts the result through down_proj. Pruning must remove corresponding positions from both expansion projections and the matching inputs to the contraction projection. section 5.1 section 6.5.3

Figure 5.1 The complete GLU structure. Information enters with dimension hidden_size (2048 in Llama-3.2-1B) and expands to intermediate_size (8192) via gate_proj and up_proj. The gate_proj output, modulated by SiLU, is element-wise multiplied with up_proj’s output. The result is contracted back to hidden_size by down_proj.

The GLU diagram shows the intermediate dimension being narrowed while the input and output dimensions remain unchanged. section 5.1

A practical pruning workflow:

  • Score each neuron pair using gate_proj and up_proj weights. One static metric sums max(weight) + |min(weight)| for the two projections; low-scoring pairs are removal candidates. section 5.3 section 5.1
  • For data-driven selection, collect L2 activation norms at down_proj’s input with PyTorch hooks, then multiply those norms by the structural scores. Check that calibration data exists for every layer. section 5.5 section 5.2.4
  • Keep the highest-scoring positions, rebuild the three projections with matching indices, and update intermediate_size in the model configuration. Copy the original model first so it remains available as a baseline. section 5.1.2 section 5.2.4
  • Test that generation works, then compare capabilities and perplexity against the baseline. Evaluate several tasks: different capabilities can degrade at different rates. section 5.1.3
Figure 5.8 Model surgery using hybrid importance. The update_model function (Listing 5.7) iterates over each layer (1-2), calls prune_neuron_pairs (3) that calculates hybrid scores by multiplying structural magnitudes by activation norms (4), selects the top-K neurons and creates new layers (5), replaces the original layers (6), and returns the specialized model (7).

This diagram traces calibration scores through neuron selection, layer reconstruction, and model replacement. section 5.2.4

Static pruning and capability trade-offs

Removing 40% of the MLP expansion neurons from Llama-3.2-1B reduced parameters by about 322 million, or 26.06%, while keeping all 16 blocks. Without recovery training, WikiText perplexity rose from 11.57 to 56.33 and GSM8K fell from 0.0660 to 0.0205. Smaller size did not preserve all capabilities equally. section 5.1.2 section 5.1.3 section 5.3

Calibration data changes which neurons survive: WikiText and SMS calibration can produce models with the same width but different retained connections. Width pruning can also precede depth pruning and knowledge distillation to recover lost knowledge. section 5.2.4 section 6.5.3

For deployment, benchmark on the target hardware. An unaligned intermediate size can cause matrix kernels to pad the width, while different widths across layers may not fit serving-engine assumptions. Alignment to hardware-friendly multiples is a useful experiment, not a guarantee of speedup. section 12.5 section 5.4

How Rearchitecting LLMs covers it

How deep it goes
Works through importance scoring, activation calibration, layer reconstruction, safety checks, and benchmarks; defers evaluation-system mechanics.
Examples
Compares static pruning of Llama-3.2-1B with WikiText- and SMS-calibrated variants.
What you can do after
Builds pruned GLU MLPs from scratch with PyTorch and evaluates their capability and parameter trade-offs.
How it connects
Connects width pruning with GLU architectures, activation norms, depth pruning, and knowledge distillation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.