Hybrid importance scores neurons for width pruning by combining structural weight importance with activation norms measured on calibration data. section 5.2
For GLU projections, the score combines two signals:
- Structural: normalized weight-range scores from gate_proj, up_proj, and down_proj. section 5.2
- Dynamic: activation norms captured at the input to down_proj, after gated information flows. section 5.2
- Multiplying the signals favors neurons that are structurally influential and active on the calibration data. section 5.2
Activation changes which neurons survive
A neuron with a structural score of 0.9 and activation of 32 survives, while neurons with low activations of 0.7 and 0.2 are removed despite comparable or higher structural scores. section 5.2
Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.
Shows where the structural and dynamic signals enter the GLU calculation and how they combine. section 5.2
