liveBook · Manning

Structural score

As explained in Rearchitecting LLMs

A structural score is a neuron's weight-based importance obtained by summing normalized contributions from its associated projection matrices.

A structural score estimates a neuron's weight-based importance for width pruning. In a GLU, it combines normalized weight-range contributions from gate_proj, up_proj, and down_proj; activation-based importance is a separate component of the hybrid score. section 5.2

Compute the structural component in three stages:

  • Measure each neuron's weight range as Max + |Min| in each projection matrix. section 5.2
  • Sum the normalized contributions across all three projections to obtain a score per neuron. section 5.2

How Rearchitecting LLMs covers it

How deep it goes
Computes Max + |Min| ranges, normalizes each projection's scores, and sums them; activation fusion is separate.
Examples
Uses gate_proj, up_proj, and down_proj; a hybrid example contrasts neurons with differing structural scores and activations.
What you can do after
Build a per-neuron weight-importance tensor from the three GLU projection matrices.
How it connects
Connects neuron pruning with GLU projections and activation-based hybrid importance.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.