liveBook · Manning

Structural importance metric

As explained in Rearchitecting LLMs

A structural importance metric measures a model component's significance based on its parameter structure or contribution to behavior.

Structural importance scores estimate which model components matter based on their weights. For GLU neuron pruning, combine each neuron's weight-based structural score with its L2 activation norm from calibration data; prune paired gate_proj and up_proj neurons together. section 5.5

Choose the scoring approach to match the optimization goal:

  • Use general structural importance metrics to prune less critical components without a specific dataset. section 1.2.5
  • Use calibration activations alongside weight magnitudes when selecting neurons for domain-sensitive pruning. section 5.5

Calibration changes what survives

Applying the same pruning percentage with WikiText versus SMS calibration retains neurons suited to formal text versus conversational language. section 5.5

How Rearchitecting LLMs covers it

How deep it goes
Combines weight magnitudes with L2 activation norms and explains paired projection pruning in GLU architectures.
Examples
Compares WikiText- and SMS-calibrated pruning, which retains neurons for different language domains.
What you can do after
Apply weight magnitudes, optionally scaled by calibration activation norms, to select neurons for width pruning.
How it connects
Connects weight-based scores to activation norms, calibration, and width pruning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.