liveBook · Manning

Feature importance score

As explained in Rearchitecting LLMs

An importance score numerically estimates how important a model component is to its information or performance.

A feature importance score estimates how much a model component contributes to information processing. For structured pruning, scores rank blocks, neuron pairs, or attention modules so that lower-scoring components can be considered for removal. The score is a selection guide, not a guarantee of minimal performance loss. section 4.3.3 section 4.3.4 section 8.3.2

Choose a score that matches the component and the available evidence:

  • For depth pruning, capture each block’s input and output during a calibration forward pass. Compare their activations with cosine similarity and aggregate the results into a block score. section 4.3.3
  • For static width pruning, score a neuron pair from the weight ranges in gate_proj and up_proj. For each projection, calculate the row maximum plus the absolute row minimum, then sum the two values. section 5.1.1
  • For data-driven width pruning, combine normalized weight-range signals from gate_proj, up_proj, and down_proj with activation norms captured at the down_proj input. The hybrid score reflects both structure and calibration behavior. section 5.2
Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.

The diagram locates the weight and activation signals used in hybrid neuron scoring. section 5.2

For block scoring, register PyTorch forward hooks, run calibration batches without gradients, and record detached inputs and outputs. Calculate cosine similarity along the hidden dimension, mask padding tokens, and aggregate valid-token results. Check for missing hook captures, empty inputs, all-padding batches, and non-finite similarities before relying on the ranking. section 4.3.3

Selecting blocks with WinoGrande

On Qwen3-0.6B, WinoGrande calibration assigns block 0 a score of 0.947 and blocks 25 and 26 scores of about 0.025 each. Selecting four blocks without protections removes [25, 26, 24, 23]. WikiText calibration instead selects [25, 24, 13, 23], demonstrating that the ranking depends on the calibration data. section 4.3.4

Sort block scores ascending to select removal candidates. Protecting the first and last blocks is a useful safeguard; avoiding adjacent removals is an optional heuristic to test, not a fixed rule. Low scores can miss subtle but critical transformations, so evaluate the resulting model on relevant tasks and compare protected and unprotected selections. section 4.3.4 section 4.5

For neuron pruning, keep the top-k hybrid scores, sort the selected indices back into their original order, and transfer the corresponding weights into reduced layers. A safety check must leave at least one neuron. Calibration coverage matters: multiplying structural and activation signals can still give a low score to a neuron that was not active on the chosen data. section 5.2.2 section 5.2

Scoring also has a memory cost. Computing scalar importance inside each hook instead of retaining full activation tensors can reduce memory use, but removes the ability to inspect those raw activations afterward. section 4.5

How Rearchitecting LLMs covers it

How deep it goes
Works through hook-based block scoring, static and hybrid neuron metrics, safety checks, and score-based selection.
Examples
Compares WinoGrande and WikiText block selections and calculates neuron scores from projection weight ranges.
What you can do after
Apply it using PyTorch hooks and top-k selection to identify pruning candidates, then evaluate the resulting models.
How it connects
Connects importance scoring to cosine similarity, activation norms, calibration data, depth pruning, and width pruning.

Also explained in

Compare how 10 books explain feature importance score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.