liveBook · Manning

Magnitude-based selection

As explained in Rearchitecting LLMs

Magnitude-based selection chooses model components for pruning according to the magnitudes of their parameters or related scores.

Magnitude-based selection ranks model components using parameter magnitudes or scores derived from them, then prunes selected components. It provides a static alternative to selection guided by data. section 4.5 section 5.3

The scoring method depends on the pruning target:

  • For width pruning, calculate each neuron's weight range in the GLU gate_proj and up_proj layers as maximum plus absolute minimum, then sum the two values into an importance score. section 5.3
  • For depth pruning, compare a static block selection, [8, 2, 7, 6], with data-driven selections and evaluate benchmark retention. section 4.5

Width-pruning trade-offs

At 40% pruning on Llama-3.2-1B, the reported method reduces expansion from 4.0x to 2.4x. Several knowledge and reasoning scores fall, while the figure illustrates improvements in instruction-following and truthfulness for Llama-3.2-3B. section 5.3

How Rearchitecting LLMs covers it

How deep it goes
Defines a neuron weight-range score and reports pruning outcomes, while block-selection exercises focus on benchmark comparisons.
Examples
Compares WinoGrande and Lambada retention for block selections and reports Llama-3.2 width-pruning results.
What you can do after
Evaluate static selections against data-guided pruning using benchmark retention and pruning-degradation curves.
How it connects
Connects structured pruning with width pruning, depth pruning, and data-driven pruning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.