liveBook · Manning

Chapter 5: Shaping model architectures via width pruning

What you will learn in chapter 5 of Rearchitecting LLMs

The chapter introduction contrasts coarse depth pruning with more precise neuron-level width pruning and identifies wide GLU-based MLPs as suitable targets.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction contrasts coarse depth pruning with more precise neuron-level width pruning and identifies wide GLU-based MLPs as suitable targets.

Key terms: depth pruning, width pruning, Transformer block, Multi-Layer Perceptron (MLP), Gated Linear Unit (GLU), wide models

5.1 Selecting static neurons

This section explains how to choose GLU MLP neurons for width pruning while preserving the model’s external dimensions and coordinated gating.

Key terms: intermediate_size, expansion, width pruning, gate_proj, up_proj, down_proj, gating synchronization, peak-to-peak magnitude range, importance metric, transformation potential

5.1.1 Identifying neurons for removal

This section shows how gate and up projection weights are organized and how a weight-range score identifies neurons for removal.

Key terms: gate_weight, up_weight, intermediate_size, hidden_size, importance score, magnitude metric, range, neuron pair

5.1.2 Reconstructing the MLP module

This section shows how to rebuild and install smaller MLP layers after selecting neuron pairs, then explains the resulting parameter savings and trade-offs.

Key terms: importance score, neuron pairs, k, torch.topk, indices_to_keep, gate_proj, up_proj, down_proj, Linear layers, layer.mlp

5.1.3 Measuring the impact: benchmarks and trade-off

This section evaluates a width-pruned model and shows that pruning damages some capabilities while improving others, creating a different trade-off rather than a uniformly worse model.

Key terms: width pruning, benchmark suite, model_evaluation, lm_eval, perplexity, GSM8K, MMLU, WinoGrande, BoolQ, IFEval

5.2 Data-driven neuron selection

The section explains how to rank GLU neurons for width pruning by combining weight-based structural scores with activations measured on calibration data.

Key terms: hybrid importance, structural score, activation norms, importance_score, calibration dataset, dynamic component, static component, domain-specific calibration data, down_proj, gate_proj

5.2.1 Extracting the dynamic importance component

This section explains how forward hooks capture and aggregate per-neuron activation norms for the dynamic component of a hybrid importance score.

Key terms: down_proj, gate_proj, SiLU, hidden_size, forward hook, X_d, L2 norm, calibration dataset, CPU storage, L1 norm

5.2.2 Reconstructing the MLP with hybrid scores

This section adapts MLP width pruning to use hybrid weight-and-activation scores, then rebuilds smaller projection layers around the selected neurons.

Key terms: X_d_norm, compute_neuron_pair_importance, hybrid importance scores, torch.topk, gate_proj, up_proj, down_proj, hidden_size, intermediate size k, calibration domain

5.2.3 Running the calibration loop

This section shows how to run calibration batches through a model so hooks collect per-neuron activation norms for data-guided pruning.

Key terms: calibration loop, down_proj, X_d, L2 norms, get_activation_norms(), evaluation mode, gradients, GPU, cache, VRAM

5.2.4 Creating domain-specialized models

The section shows how calibration data guides hybrid width pruning to produce same-sized models with different preserved neurons and domain-specific behavior.

Key terms: activation_norms_wiki, activation_norms_sms, calibration data, narrower MLP, update_model, activation norm tensor, prune_neuron_pairs, hybrid importance scores, model configuration, 20% pruning

5.2.5 Measuring the specialization effect

This section uses perplexity results to show how calibration data shapes width-pruned models for particular text domains, while explaining the limits of pruning without recovery.

Key terms: calibration dataset, expansion neurons, perplexity, wiki_model, sms_model, static criterion, weight magnitudes, data-driven method, base model, data-driven pruning

5.2.6 Inference performance and energy efficiency

This section compares the inference speed and energy cost of width-pruned models, then explains when GPU bottlenecks and matrix-dimension alignment affect the gains.

Key terms: calibration dataset, percentage of pruning, tokens per second, matrix dimensions, joules, energy consumed per generated token, expansion reduction, comparative baseline, expansion ratio, memory movements

5.3 From paper to practice

This section connects width-pruning results to two research papers and explains why the chapter combines structural weight scores with data-specific activations rather than copying either paper’s method.

Key terms: gate_proj, up_proj, structural range, importance score, width pruning dichotomy, perplexity, IFEval, reasoning capabilities, CFSP, coarse-to-fine

5.4 Hands-on lab

5.5 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 4: Building smaller and faster LLMs with depth pruning Chapter 6: Knowledge recovery through distillation →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.