A hybrid importance score ranks neurons for width pruning by combining structural weight magnitude with observed activation. Multiplying the two favors neurons that have both substantial weights and high activity; a low value in either component lowers the score. section 5.2
For a GLU MLP, the calculation has three stages:
- Measure each neuron’s weight range across gate_proj, up_proj, and down_proj using maximum plus absolute minimum; use rows for the first two matrices and columns for down_proj. section 5.2
- Normalize each projection’s range to [0, 1] before combining contributions, so differences in weight scale do not let one projection dominate. section 5.2
- Multiply the structural measure by accumulated activation norms captured at the input to down_proj. Neurons with weak activation can score poorly even when their weights are large. section 5.2
Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.
The figure locates the activation measurement and shows how it combines with weight ranges to produce the score. section 5.2
Using scores for pruning
During domain-specialized pruning, accumulated activation norms are supplied for each layer and combined with structural scores to select neurons. A 20% pruning setting reduces each MLP’s intermediate width from 8192 to 6554; the preserved neurons can differ with the calibration data. section 5.2.4
