liveBook · Manning

Dynamic importance component

As explained in Rearchitecting LLMs

Dynamic importance is a measure of a model component's significance derived from its observed activations on representative inputs.

Dynamic importance measures how strongly intermediate neurons activate on calibration inputs. In width pruning, activation norms complement weight-based structural importance, favoring neurons that are both structurally significant and actively used. section 5.2.1 section 5.5

Forward hooks capture activations at the input to down_proj, after gating and before dimensionality reduction. They compute L2 norms and accumulate results on the CPU to limit GPU memory use during offline calibration. section 5.2.1 section 5.5

Listing 5.4 Configuring hooks to capture MLP activation norms
_accumulated_act_norms = {} #1
 
def setup_mlp_hooks_for_importance(model, device):
    global _accumulated_act_norms
    _accumulated_act_norms.clear() #2
    gc.collect() #2
    torch.cuda.empty_cache() #2
    handles = []
 
    for idx, layer in enumerate(model.model.layers):
        intermediate_size = layer.mlp.down_proj.in_features
        _accumulated_act_norms[idx] = torch.zeros(
            intermediate_size,
            dtype=torch.float32,
            device='cpu' #3
        )
 
    def make_hook(layer_idx): #4
        def hook(module, input, output): #5
 
            X_d = input[0].detach() #6
 
            act_norms_L2 = torch.norm(
                X_d.to(torch.float32),
                p=2,
                dim=(0, 1)  #7
            )
            _accumulated_act_norms[layer_idx] +=
 ➥act_norms_L2.cpu() #8
 
       return hook
 
    for idx, layer in enumerate(model.model.layers):
        handle = layer.mlp.down_proj.register_forward_hook(
            make_hook(idx)
        ) #9
        handles.append(handle)
 
    print(f"✓ Registered {len(handles)} hooks on down_proj")
 
    return handles
 
def get_activation_norms():
    return {
        layer_idx: norms.clone()
        for layer_idx, norms in _accumulated_act_norms.items()
    }

The listing shows how to initialize per-layer storage and configure hooks to accumulate MLP activation norms. section 5.2.1

Calibration data changes which neurons survive

Applying the same pruning percentage with WikiText versus SMS calibration preserves neurons suited to formal text versus conversational language. section 5.5

How Rearchitecting LLMs covers it

How deep it goes
Explains hook placement, L2-norm calculation, and CPU accumulation during calibration.
Examples
Compares WikiText and SMS calibration, which preserve neurons for formal and conversational language respectively.
What you can do after
Apply via PyTorch forward hooks to capture and accumulate MLP activation norms for calibration.
How it connects
Connects width pruning with structural importance, activation functions, and PyTorch forward hooks.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.