liveBook · Manning

Hybrid importance score

As explained in Rearchitecting LLMs

A hybrid importance score combines activation norms with a structural measure such as weight magnitude to assess neuron importance.

A hybrid importance score ranks neurons for width pruning by combining structural weight magnitude with observed activation. Multiplying the two favors neurons that have both substantial weights and high activity; a low value in either component lowers the score. section 5.2

For a GLU MLP, the calculation has three stages:

  • Measure each neuron’s weight range across gate_proj, up_proj, and down_proj using maximum plus absolute minimum; use rows for the first two matrices and columns for down_proj. section 5.2
  • Normalize each projection’s range to [0, 1] before combining contributions, so differences in weight scale do not let one projection dominate. section 5.2
  • Multiply the structural measure by accumulated activation norms captured at the input to down_proj. Neurons with weak activation can score poorly even when their weights are large. section 5.2
Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.

The figure locates the activation measurement and shows how it combines with weight ranges to produce the score. section 5.2

Using scores for pruning

During domain-specialized pruning, accumulated activation norms are supplied for each layer and combined with structural scores to select neurons. A 20% pruning setting reduces each MLP’s intermediate width from 8192 to 6554; the preserved neurons can differ with the calibration data. section 5.2.4

How Rearchitecting LLMs covers it

How deep it goes
Calculates normalized weight ranges across three GLU projections and multiplies their structural contribution by activation norms.
Examples
Uses GLU projection weights and activation norms to prune 20% of neurons in domain-specialized models.
What it assumes
Requires GLU projection weights and per-layer activation norms; captures activation data through PyTorch hooks.
What you can do after
Apply the score in a layer-pruning workflow to select neurons using both structural weights and calibration activations.
How it connects
Connects neuron pruning and activation norms with GLU projections and calibration data.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.