Data-driven pruning reduces model depth or width by measuring component importance on calibration inputs representative of the intended task. Unlike fixed position or weight-magnitude rules, it uses observed behavior to help preserve capabilities needed in a particular domain. section 4.2.3 section 13.4.1
A practical pruning workflow has four stages:
- Choose calibration inputs that match the intended use. For a tool-using specialist, use training prompts containing the system message, tools, and user query; tokenize, truncate, and batch them with padding and attention masks. section 13.4.1
- Measure importance. For depth pruning, rank Transformer blocks by how much they transform their inputs; cosine distance can identify relatively passive blocks. For width pruning, combine neuron activation measurements with weight information. section 13.4.1
- Remove low-ranked components subject to chosen protections. For blocks, these can preserve initial and final layers and prevent adjacent removals. For MLP width, retain the highest-scoring neuron pairs and reconstruct the projections consistently. section 13.4.1 section 5.2.2
- Evaluate against the base model and static-pruning baselines on both the target task and other relevant benchmarks. Compare different pruning amounts and protection settings before committing resources to recovery. section 4.5 section 6.1
Figure 13.3 Selection of transformer blocks for depth pruning, applied to the fine-tuned specialist T (28 blocks). Row 1 ranks all blocks by passivity (cosine distance between each block's input and output) and marks the three most passive blocks — 23, 25, and 26 — as the unconstrained selection. Row 2 applies heuristic protection over blocks 0–3 and 26–27: block 26 is excluded, and block 24 takes its place as the next most passive candidate. Row 3 applies adjacency protection: block 24 is excluded for being adjacent to both 23 and 25, and block 13 takes its place.

The selection example shows how protected and adjacent blocks are excluded from an importance-ranked list. section 13.4.1
Depth pruning with WinoGrande
Removing four blocks using WinoGrande cosine importance selected [25, 26, 24, 23]; adding final-layer and adjacency protections selected [25, 23, 13, 15]. Against base scores of 0.566 on WinoGrande and 0.404 on Lambada, the protected model scored 0.550 and 0.2954; the unprotected model scored 0.545 and 0.0413. The two benchmarks expose different levels of degradation. section 4.3.5
Width pruning for a domain
At approximately 1.07B parameters, a WikiText-calibrated model had perplexity 36.28 on WikiText and 182.54 on SMS; an SMS-calibrated model scored 48.65 and 165.54. Each performed better on its calibration domain, but both had higher perplexity than the 1.24B-parameter base model. section 5.2.5
Calibration directs what the model preserves, so a dataset suited to one task may sacrifice performance on another. Protection heuristics and data-guided rankings are candidates to test, not guarantees; evaluate the quality and efficiency trade-off before deployment or distillation. section 5.2.5 section 6.1
