A feature importance score estimates how much a model component contributes to information processing. For structured pruning, scores rank blocks, neuron pairs, or attention modules so that lower-scoring components can be considered for removal. The score is a selection guide, not a guarantee of minimal performance loss. section 4.3.3 section 4.3.4 section 8.3.2
Choose a score that matches the component and the available evidence:
- For depth pruning, capture each block’s input and output during a calibration forward pass. Compare their activations with cosine similarity and aggregate the results into a block score. section 4.3.3
- For static width pruning, score a neuron pair from the weight ranges in gate_proj and up_proj. For each projection, calculate the row maximum plus the absolute row minimum, then sum the two values. section 5.1.1
- For data-driven width pruning, combine normalized weight-range signals from gate_proj, up_proj, and down_proj with activation norms captured at the down_proj input. The hybrid score reflects both structure and calibration behavior. section 5.2
Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.
The diagram locates the weight and activation signals used in hybrid neuron scoring. section 5.2
For block scoring, register PyTorch forward hooks, run calibration batches without gradients, and record detached inputs and outputs. Calculate cosine similarity along the hidden dimension, mask padding tokens, and aggregate valid-token results. Check for missing hook captures, empty inputs, all-padding batches, and non-finite similarities before relying on the ranking. section 4.3.3
Selecting blocks with WinoGrande
On Qwen3-0.6B, WinoGrande calibration assigns block 0 a score of 0.947 and blocks 25 and 26 scores of about 0.025 each. Selecting four blocks without protections removes [25, 26, 24, 23]. WikiText calibration instead selects [25, 24, 13, 23], demonstrating that the ranking depends on the calibration data. section 4.3.4
Sort block scores ascending to select removal candidates. Protecting the first and last blocks is a useful safeguard; avoiding adjacent removals is an optional heuristic to test, not a fixed rule. Low scores can miss subtle but critical transformations, so evaluate the resulting model on relevant tasks and compare protected and unprotected selections. section 4.3.4 section 4.5
For neuron pruning, keep the top-k hybrid scores, sort the selected indices back into their original order, and transfer the corresponding weights into reduced layers. A safety check must leave at least one neuron. Calibration coverage matters: multiplying structural and activation signals can still give a low score to a neuron that was not active on the chosen data. section 5.2.2 section 5.2
Scoring also has a memory cost. Computing scalar importance inside each hook instead of retaining full activation tensors can reduce memory use, but removes the ability to inspect those raw activations afterward. section 4.5









