Chapter 5: Shaping model architectures via width pruning
The chapter introduction contrasts coarse depth pruning with more precise neuron-level width pruning and identifies wide GLU-based MLPs as suitable targets.
After this chapter you will be able to
- Depth versus width pruning. Distinguish depth pruning, which removes an entire Transformer block, from width pruning, which targets specific neurons in a GLU-based MLP. Explain that neuron-level targeting is more precise, is suited to wide models, and may affect both performance and model behavior.
- MLP expansion width. Explain why width pruning changes the MLP’s intermediate_size rather than the size passed between transformer blocks.
- Coordinated GLU neuron pairs. Describe how gate_proj and up_proj work together and why their corresponding neurons must be removed together.
- Paired peak-to-peak scoring. Explain how the proposed static importance score identifies neuron pairs as pruning candidates.
- Weight-range importance ranking. Explain how the section scores a neuron from its row of weights and how that score identifies a removal candidate.
- Neuron pair selection. Trace how the function turns neuron-pair importance scores and a pruning percentage into the indices retained for a smaller intermediate size.
- Reduced MLP weight transfer. Describe the dimensions of the replacement MLP layers and how the retained indices are used to copy weights into them.
- Model-wide architecture update. Trace how the pruning function is applied to each block and explain why the model configuration must also be updated.
- Width pruning structural effects. Distinguish the demonstrated parameter reduction from removing complete blocks, and state what remains unchanged.
- Multidimensional pruning evaluation. Explain why a functional text-generation example is not enough to evaluate width pruning and why the section compares the base and pruned models across several kinds of benchmarks.
- Uneven capability losses. Describe how the pruned model's fluency and reasoning compare with its more resilient knowledge and basic language comprehension.
- Unexpected precision gains. Explain which benchmarks improved after pruning and why those results complicate the claim that the pruned model is simply worse.
- Hybrid neuron importance. Explain how the hybrid importance score combines a neuron's structural score with its activation norm, and how that changes which neurons are kept.
- Per-neuron activation capture. Explain why activations are measured at the input to down_proj rather than at its output.
- Normalized three-projection score. Describe how the weight-based structural score is calculated across the three GLU projections and why their contributions are normalized.
- Preprojection neuron capture point. Explain why activation is captured at the input to down_proj, after gating, rather than at its output.
- Hook-based norm accumulation. Trace how layer-specific forward hooks turn calibration activations into accumulated per-neuron scores without retaining raw activations on the GPU.
- Hybrid neuron selection. Explain how the pruning function uses weight matrices and calibration activation norms to rank neuron pairs, then selects the neurons to keep.
- Reduced MLP layer reconstruction. Describe the dimensions of the replacement MLP projections and how weights for the selected neuron pairs are copied into them.
- Activation capture calibration loop. Trace how the calibration loop registers hooks, runs batches through the model, and retrieves per-layer neuron importance scores.
- Calibration-dependent neuron selection. Explain why pruning the same base model with WikiText versus SMS activation norms preserves different neurons even when both models have the same narrower MLP architecture.
- Per-layer hybrid model update. Trace how update_model uses per-layer activation norms to reconstruct the MLP and keep the model configuration compatible.
- Observed domain specialization. Describe how the example completions illustrate different behavior after WikiText- and SMS-calibrated pruning.
- Calibration-directed domain specialization. Explain how two models pruned by the same amount retain different neurons according to their calibration data, and how perplexity shows each model's advantage on its own text domain.
- Capacity loss and recovery. Explain why successful on-domain specialization does not make the pruned models better than the original base model, and state the intended role of later recovery.
- Pruning Amount and Inference Speed. Explain why models pruned using different calibration datasets can have the same inference-speed improvement even though their capability outcomes differ.
- Energy Cost and Output Behavior. Distinguish reduced energy per generated token from changes in how many tokens the pruned model generates, and explain how cautiously to interpret the energy measurements.
- Limits of Width-Pruning Speedups. Explain why reducing MLP expansion does not always translate into substantial inference-speed gains.
- Uneven capability effects of pruning. Describe the reported width-pruning dichotomy: knowledge, reasoning, and text prediction can deteriorate substantially while some instruction-following measures improve, without assuming a perfectly linear trend.
- Hybrid structural activation score. Distinguish the chapter's hybrid neuron score from CFSP's full coarse-to-fine formula, and explain why multiplying structural scores by captured activations can flag seldom-used neurons for removal.
- Selective adaptation of research methods. Justify why the chapter draws different ideas from the two papers instead of implementing either paper unchanged.
What it builds on
- How depth pruning works (chapter 4) · Depth pruning as removal of Transformer blocks.
- The evolution of the MLP: from simple expansion to Gated Linear Units (GLU) (chapter 3) · The GLU structure of a modern MLP.
- The Transformer dimensions: depth and width (chapter 3) · The distinction between transformer depth and width.
- Removing by weight (chapter 4) · Selecting pruning candidates using weight magnitude.
- Using PyTorch hooks (chapter 4) · Introduces PyTorch hooks for capturing activations during forward passes.
Section by section
Introduction
The chapter introduction contrasts coarse depth pruning with more precise neuron-level width pruning and identifies wide GLU-based MLPs as suitable targets.
- Depth versus width pruning: Distinguish depth pruning, which removes an entire Transformer block, from width pruning, which targets specific neurons in a GLU-based MLP. Explain that neuron-level targeting is more precise, is suited to wide models, and may affect both performance and model behavior.
Why it matters: The distinction explains why this chapter moves from removing whole blocks to targeting neurons within a block.
Key terms: depth pruning, width pruning, Transformer block, Multi-Layer Perceptron (MLP), Gated Linear Unit (GLU), wide models
5.1 Selecting static neurons
This section explains how to choose GLU MLP neurons for width pruning while preserving the model’s external dimensions and coordinated gating.
- MLP expansion width: Explain why width pruning changes the MLP’s intermediate_size rather than the size passed between transformer blocks.
Why it matters: Keeping pruning internal to the MLP avoids breaking the connection to the next block. - Coordinated GLU neuron pairs: Describe how gate_proj and up_proj work together and why their corresponding neurons must be removed together.
Why it matters: The unit selected for pruning is a coordinated pair, not an isolated projection neuron. - Paired peak-to-peak scoring: Explain how the proposed static importance score identifies neuron pairs as pruning candidates.
Why it matters: Scoring both sides of the pair avoids discarding a neuron whose contribution appears small in only one projection. - SiLU gating activation: Describe SiLU’s role in the GLU MLP at the level explained here.
Why it matters: SiLU helps explain why the GLU gate does more than simply pass signals through unchanged.
Key terms: intermediate_size, expansion, width pruning, gate_proj, up_proj, down_proj, gating synchronization, peak-to-peak magnitude range, importance metric, transformation potential
5.1.1 Identifying neurons for removal
This section shows how gate and up projection weights are organized and how a weight-range score identifies neurons for removal.
- Projection weights and neuron rows: Describe how the gate and up projection weight matrices are laid out: each row corresponds to an intermediate-layer neuron, and each column corresponds to an input dimension from the hidden state.
Why it matters: The matrix layout tells you which weights belong to each neuron being considered for pruning. - Weight-range importance ranking: Explain how the section scores a neuron from its row of weights and how that score identifies a removal candidate.
Why it matters: The score turns each neuron's weights into a value that can be used to select candidates for width pruning.
Key terms: gate_weight, up_weight, intermediate_size, hidden_size, importance score, magnitude metric, range, neuron pair
5.1.2 Reconstructing the MLP module
This section shows how to rebuild and install smaller MLP layers after selecting neuron pairs, then explains the resulting parameter savings and trade-offs.
- Neuron pair selection: Trace how the function turns neuron-pair importance scores and a pruning percentage into the indices retained for a smaller intermediate size.
Why it matters: The retained indices determine both the MLP’s new width and which weights must move into its replacement layers. - Reduced MLP weight transfer: Describe the dimensions of the replacement MLP layers and how the retained indices are used to copy weights into them.
Why it matters: Matching the selected features across all three projections makes the narrower MLP connect correctly to the rest of the block. - Model-wide architecture update: Trace how the pruning function is applied to each block and explain why the model configuration must also be updated.
Why it matters: Changing layer weights without changing the recorded dimensions can cause downstream code to misread or fail to load the model. - Pruning percentage trade-off: Explain why the example uses 40% pruning and how the section recommends choosing a pruning percentage in practice.
Why it matters: The example’s 40% setting illustrates an architectural choice, not a universally safe pruning rate. - Width pruning structural effects: Distinguish the demonstrated parameter reduction from removing complete blocks, and state what remains unchanged.
Why it matters: A large parameter saving does not mean width pruning has the same structural or memory effects as depth pruning.
Key terms: importance score, neuron pairs, k, torch.topk, indices_to_keep, gate_proj, up_proj, down_proj, Linear layers, layer.mlp
5.1.3 Measuring the impact: benchmarks and trade-off
This section evaluates a width-pruned model and shows that pruning damages some capabilities while improving others, creating a different trade-off rather than a uniformly worse model.
- Multidimensional pruning evaluation: Explain why a functional text-generation example is not enough to evaluate width pruning and why the section compares the base and pruned models across several kinds of benchmarks.
Why it matters: A model that still answers prompts may nevertheless have lost abilities that matter for its intended use. - Uneven capability losses: Describe how the pruned model's fluency and reasoning compare with its more resilient knowledge and basic language comprehension.
Why it matters: The losses identify tasks for which this smaller model would be a poor substitute for the base model. - Unexpected precision gains: Explain which benchmarks improved after pruning and why those results complicate the claim that the pruned model is simply worse.
Why it matters: Benchmark gains can reveal a useful change in behavior even when overall sophistication declines. - Precision-oriented application fit: Apply the section's capability trade-off to tasks where strict adherence to an output constraint matters more than creative elaboration.
Why it matters: Model selection should match the changed capability profile to the application's requirements. - Weight-magnitude selection limitation: Explain the limitation of using weight magnitude alone to select neurons for pruning.
Why it matters: This limitation explains why a pruning choice based only on weights may preserve the wrong capabilities for an intended use.
Key terms: width pruning, benchmark suite, model_evaluation, lm_eval, perplexity, GSM8K, MMLU, WinoGrande, BoolQ, IFEval
5.2 Data-driven neuron selection
The section explains how to rank GLU neurons for width pruning by combining weight-based structural scores with activations measured on calibration data.
- Hybrid neuron importance: Explain how the hybrid importance score combines a neuron's structural score with its activation norm, and how that changes which neurons are kept.
Why it matters: It explains why calibration data can change a pruning decision even when two neurons have identical weights. - Calibration data trade-off: Explain why the section combines weight information with activations rather than relying only on activations from a domain-specific calibration dataset.
Why it matters: Calibration activity is informative, but it is not a complete picture of a neuron's value. - Per-neuron activation capture: Explain why activations are measured at the input to down_proj rather than at its output.
Why it matters: Measuring before the signals are mixed makes the dynamic score attributable to an individual neuron. - Normalized three-projection score: Describe how the weight-based structural score is calculated across the three GLU projections and why their contributions are normalized.
Why it matters: Comparable contributions from all three projections make the structural part of the hybrid score meaningful.
Key terms: hybrid importance, structural score, activation norms, importance_score, calibration dataset, dynamic component, static component, domain-specific calibration data, down_proj, gate_proj
5.2.1 Extracting the dynamic importance component
This section explains how forward hooks capture and aggregate per-neuron activation norms for the dynamic component of a hybrid importance score.
- Preprojection neuron capture point: Explain why activation is captured at the input to down_proj, after gating, rather than at its output.
Why it matters: The measurement point determines whether the score can be attributed to individual intermediate neurons. - Hook-based norm accumulation: Trace how layer-specific forward hooks turn calibration activations into accumulated per-neuron scores without retaining raw activations on the GPU.
Why it matters: Calculating compact statistics inside the hooks makes dataset-level neuron scoring practical without storing every activation. - L2 scoring rationale: Distinguish L2 norms from signed sums and L1 norms as measures of neuron activity, and explain why L2 favors activation spikes.
Why it matters: The choice of norm affects which neurons appear important enough to preserve.
Key terms: down_proj, gate_proj, SiLU, hidden_size, forward hook, X_d, L2 norm, calibration dataset, CPU storage, L1 norm
5.2.2 Reconstructing the MLP with hybrid scores
This section adapts MLP width pruning to use hybrid weight-and-activation scores, then rebuilds smaller projection layers around the selected neurons.
- Hybrid neuron selection: Explain how the pruning function uses weight matrices and calibration activation norms to rank neuron pairs, then selects the neurons to keep.
Why it matters: The ranking determines which neuron pairs survive the smaller MLP. - Reduced MLP layer reconstruction: Describe the dimensions of the replacement MLP projections and how weights for the selected neuron pairs are copied into them.
Why it matters: Pruning only works as a smaller, compatible MLP if the replacement dimensions and copied weight axes match. - Calibration domain preservation: Explain why the choice of calibration data affects which capabilities the pruned model is intended to preserve.
Why it matters: Choosing calibration data lets you steer which kinds of activity influence the neurons retained.
Key terms: X_d_norm, compute_neuron_pair_importance, hybrid importance scores, torch.topk, gate_proj, up_proj, down_proj, hidden_size, intermediate size k, calibration domain
5.2.3 Running the calibration loop
This section shows how to run calibration batches through a model so hooks collect per-neuron activation norms for data-guided pruning.
- Activation capture calibration loop: Trace how the calibration loop registers hooks, runs batches through the model, and retrieves per-layer neuron importance scores.
Why it matters: The loop turns calibration data into the neuron importance scores used for guided pruning. - Calibration execution and memory choices: Explain why the loop uses evaluation mode, disables gradients, places inputs on the GPU, and may periodically clear the cache.
Why it matters: These settings make activation measurement consistent while keeping calibration practical on limited GPU memory. - Dataset-specific activation norms: Describe how the same calibration loop produces separate norm dictionaries for WikiText-2 and SMS Spam data.
Why it matters: Separate calibration data gives the pruning process separate importance scores for the two text domains.
Key terms: calibration loop, down_proj, X_d, L2 norms, get_activation_norms(), evaluation mode, gradients, GPU, cache, VRAM
5.2.4 Creating domain-specialized models
The section shows how calibration data guides hybrid width pruning to produce same-sized models with different preserved neurons and domain-specific behavior.
- Calibration-dependent neuron selection: Explain why pruning the same base model with WikiText versus SMS activation norms preserves different neurons even when both models have the same narrower MLP architecture.
Why it matters: The calibration dataset can change what a pruned model retains without changing its size. - Per-layer hybrid model update: Trace how update_model uses per-layer activation norms to reconstruct the MLP and keep the model configuration compatible.
Why it matters: The update links calibration measurements to the model’s physical structure while avoiding missing-data and configuration problems. - Conservative pruning comparison: Justify the section’s choice of 20% pruning and state its effect on MLP width.
Why it matters: A less aggressive, equal reduction makes the calibration-data comparison easier to interpret. - Observed domain specialization: Describe how the example completions illustrate different behavior after WikiText- and SMS-calibrated pruning.
Why it matters: The example connects a change in retained neurons to a visible change in model output.
Key terms: activation_norms_wiki, activation_norms_sms, calibration data, narrower MLP, update_model, activation norm tensor, prune_neuron_pairs, hybrid importance scores, model configuration, 20% pruning
5.2.5 Measuring the specialization effect
This section uses perplexity results to show how calibration data shapes width-pruned models for particular text domains, while explaining the limits of pruning without recovery.
- Calibration-directed domain specialization: Explain how two models pruned by the same amount retain different neurons according to their calibration data, and how perplexity shows each model's advantage on its own text domain.
Why it matters: It shows that pruning can preserve capabilities for a particular kind of text rather than merely make a model smaller. - Static baseline and task metrics: Distinguish the observed perplexity results for data-driven pruning from a general guarantee that it beats static pruning on every metric or task.
Why it matters: A metric should be interpreted against the task the pruning was meant to preserve. - Capacity loss and recovery: Explain why successful on-domain specialization does not make the pruned models better than the original base model, and state the intended role of later recovery.
Why it matters: It keeps the specialization result in perspective: targeted pruning is a starting point for an efficient recovered model, not a claim that capacity loss is free.
Key terms: calibration dataset, expansion neurons, perplexity, wiki_model, sms_model, static criterion, weight magnitudes, data-driven method, base model, data-driven pruning
5.2.6 Inference performance and energy efficiency
This section compares the inference speed and energy cost of width-pruned models, then explains when GPU bottlenecks and matrix-dimension alignment affect the gains.
- Pruning Amount and Inference Speed: Explain why models pruned using different calibration datasets can have the same inference-speed improvement even though their capability outcomes differ.
Why it matters: It separates the choice of neurons for model capability from the architectural change responsible for the measured speedup. - Energy Cost and Output Behavior: Distinguish reduced energy per generated token from changes in how many tokens the pruned model generates, and explain how cautiously to interpret the energy measurements.
Why it matters: A lower cost per token is useful for deployment, but it should not be confused with unchanged generation behavior or a precise absolute energy measurement. - Limits of Width-Pruning Speedups: Explain why reducing MLP expansion does not always translate into substantial inference-speed gains.
Why it matters: It helps predict when a smaller model will actually run faster rather than merely require less compute and memory. - GPU Matrix Dimension Alignment: Explain why aligning the pruned intermediate size to GPU-friendly multiples can improve performance in compute-bound settings but may not help on a T4.
Why it matters: It explains why choosing a slightly different retained width can matter on a busy compute-bound GPU without promising a speedup on every device.
Key terms: calibration dataset, percentage of pruning, tokens per second, matrix dimensions, joules, energy consumed per generated token, expansion reduction, comparative baseline, expansion ratio, memory movements
5.3 From paper to practice
This section connects width-pruning results to two research papers and explains why the chapter combines structural weight scores with data-specific activations rather than copying either paper’s method.
- Structural neuron importance scores: Explain how the static method scores a GLU neuron from its gate_proj and up_proj weight ranges, and why a larger range suggests greater capacity to transform activations.
Why it matters: This explains what the static score captures before activations are added to it. - Uneven capability effects of pruning: Describe the reported width-pruning dichotomy: knowledge, reasoning, and text prediction can deteriorate substantially while some instruction-following measures improve, without assuming a perfectly linear trend.
Why it matters: A single improved benchmark can hide serious damage to other model capabilities. - Hybrid structural activation score: Distinguish the chapter's hybrid neuron score from CFSP's full coarse-to-fine formula, and explain why multiplying structural scores by captured activations can flag seldom-used neurons for removal.
Why it matters: The hybrid score considers both what a neuron could do and whether it is used on the relevant data. - Selective adaptation of research methods: Justify why the chapter draws different ideas from the two papers instead of implementing either paper unchanged.
Why it matters: The example shows how to turn published findings into a method suited to a practical task.
Key terms: gate_proj, up_proj, structural range, importance score, width pruning dichotomy, perplexity, IFEval, reasoning capabilities, CFSP, coarse-to-fine
5.4 Hands-on lab
5.5 Summary
Where this is used later
- Beyond the benchmarks (chapter 8) · The distinction between pruning model width and removing other components.
- The padding tax (chapter 12) · The feed-forward width can be reduced by removing neurons.
- Combining with width pruning (chapter 6) · Reconstructing an MLP after removing neurons.
- Will it load: from research artifact to servable model (chapter 12) · Width pruning changes the MLP shape.
- Selecting the best candidate (chapter 6) · The quality losses inform selection of a pruning candidate for knowledge recovery.
- Pruning the specialist (chapter 13) · Hybrid scores combine neuron weights with data-driven information.
Concepts explained in this chapter
- width pruningWidth pruning is a structured pruning technique that removes selected neurons or channels to reduce a model's width.
- activation normAn activation norm is a numerical magnitude of a neural activation signal, used to quantify how strongly a unit responds to input data.
- down projection layerA down projection is a linear transformation that reduces an intermediate representation back to the model's hidden dimension.
- intermediate sizeAn intermediate size is the dimensionality of a neural module's intermediate representation, such as the expansion dimension of an MLP.
- feature importance scoreAn importance score numerically estimates how important a model component is to its information or performance.
- gate projectionA gate projection produces the signal that controls another intermediate signal in a gated feed-forward block.
- up projectionAn up projection is a feed-forward linear transformation that expands a hidden representation into an intermediate dimension.
- hookA model hook is a callback attached to a neural-network component that observes or modifies computation at a specified point during execution.
- data-driven selectionData-driven selection is the choice of model components based on their measured behavior on task-relevant data.
- hybrid importance scoreA hybrid importance score combines activation norms with a structural measure such as weight magnitude to assess neuron importance.
- multilayer perceptronA multilayer perceptron is a feed-forward neural network composed of successive learned linear transformations and nonlinear activations.
- l2 normThe L2 norm is the square root of the sum of squared vector components, measuring vector magnitude with greater emphasis on large components.
- activation captureActivation capture records intermediate neural signals produced during model execution.
- dynamic importance componentDynamic importance is a measure of a model component's significance derived from its observed activations on representative inputs.
- calibration dataCalibration data is representative input data used to measure model behavior and guide data-dependent model compression decisions.
- specializationDomain specialization is the adaptation of a general model to perform effectively on a particular domain or task.
- hybrid importanceHybrid importance is a neuron-selection technique that multiplies normalized weight-based structural importance by activation magnitude measured on calibration data.
- structural scoreA structural score is a neuron's weight-based importance obtained by summing normalized contributions from its associated projection matrices.
- data-driven methodData-driven pruning selects model components for removal using measurements from representative input data.
- gated linear unitA gated linear unit is a neural-network structure that modulates one linear transformation with a learned gate derived from another transformation.
- neuron pairA neuron pair is the corresponding pair of intermediate MLP neurons represented across the gated projection paths.
- torch.topktorch.topk is a PyTorch operation that returns the largest or smallest k elements of a tensor together with their indices.
- neuronA neuron is a computational unit in a neural network that transforms inputs using learned parameters and a nonlinear activation.
- Tensor CoreA Tensor Core is a specialized NVIDIA GPU execution unit optimized for high-throughput matrix arithmetic.
- dimension alignmentDimension alignment is the adjustment of tensor or matrix dimensions to values that match hardware processing constraints efficiently.
- energy consumptionEnergy consumption is the quantity of electrical energy used by a computation or hardware system.
- energy efficiencyEnergy efficiency is the amount of energy required to perform a computational workload or produce a unit of model output.
- inference performanceInference speed is the rate at which a trained model produces outputs from inputs.
- throughputThroughput is the amount of model work or output that a system completes per unit of time.
- instruction followingInstruction following is the ability of a model to carry out requested operations while satisfying specified constraints.
- magnitude-based selectionMagnitude-based selection chooses model components for pruning according to the magnitudes of their parameters or related scores.
- perplexityPerplexity is a language-modeling metric that measures how well a model predicts a sequence, with lower values indicating better predictive performance.
- recoveryA recovery process is a post-restructuring procedure that restores or improves a model’s performance after substantial structural components have been removed.
- static pruningStatic pruning selects model components for removal using fixed structural information or parameter statistics rather than runtime activation behavior.
- SwishSiLU is a smooth activation function defined by multiplying an input by its sigmoid.
- structural importance metricA structural importance metric measures a model component's significance based on its parameter structure or contribution to behavior.
- activationAn activation is a value produced by a neural-model unit during computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
