liveBook · Manning

Hybrid importance

As explained in Rearchitecting LLMs

Hybrid importance is a neuron-selection technique that multiplies normalized weight-based structural importance by activation magnitude measured on calibration data.

Hybrid importance scores neurons for width pruning by combining structural weight importance with activation norms measured on calibration data. section 5.2

For GLU projections, the score combines two signals:

  • Structural: normalized weight-range scores from gate_proj, up_proj, and down_proj. section 5.2
  • Dynamic: activation norms captured at the input to down_proj, after gated information flows. section 5.2
  • Multiplying the signals favors neurons that are structurally influential and active on the calibration data. section 5.2

Activation changes which neurons survive

A neuron with a structural score of 0.9 and activation of 32 survives, while neurons with low activations of 0.7 and 0.2 are removed despite comparable or higher structural scores. section 5.2

Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.

Shows where the structural and dynamic signals enter the GLU calculation and how they combine. section 5.2

How Rearchitecting LLMs covers it

How deep it goes
Combines normalized projection weight-range scores with activation norms captured at the down_proj input.
Examples
Contrasts neurons with activations of 32, 0.7, and 0.2 to show how activity affects survival.
What it assumes
Assumes familiarity with a weight-rank metric and calibration datasets prepared elsewhere.
What you can do after
Apply it to score GLU neurons using projection weights and activation norms from calibration data.
How it connects
Links neuron pruning to calibration data, activation norms, weight-based importance, and GLU projections.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.