Structural importance scores estimate which model components matter based on their weights. For GLU neuron pruning, combine each neuron's weight-based structural score with its L2 activation norm from calibration data; prune paired gate_proj and up_proj neurons together. section 5.5
Choose the scoring approach to match the optimization goal:
- Use general structural importance metrics to prune less critical components without a specific dataset. section 1.2.5
- Use calibration activations alongside weight magnitudes when selecting neurons for domain-sensitive pruning. section 5.5
Calibration changes what survives
Applying the same pruning percentage with WikiText versus SMS calibration retains neurons suited to formal text versus conversational language. section 5.5
