Magnitude-based selection ranks model components using parameter magnitudes or scores derived from them, then prunes selected components. It provides a static alternative to selection guided by data. section 4.5 section 5.3
The scoring method depends on the pruning target:
- For width pruning, calculate each neuron's weight range in the GLU gate_proj and up_proj layers as maximum plus absolute minimum, then sum the two values into an importance score. section 5.3
- For depth pruning, compare a static block selection, [8, 2, 7, 6], with data-driven selections and evaluate benchmark retention. section 4.5
Width-pruning trade-offs
At 40% pruning on Llama-3.2-1B, the reported method reduces expansion from 4.0x to 2.4x. Several knowledge and reasoning scores fall, while the figure illustrates improvements in instruction-following and truthfulness for Llama-3.2-3B. section 5.3
