liveBook · Manning

L2 norm

As explained in Rearchitecting LLMs

The L2 norm is the square root of the sum of squared vector components, measuring vector magnitude with greater emphasis on large components.

In width pruning, the L2 norm measures activation intensity: square the components, sum them, and take the square root. Applied to [B, S, I] activations across batch and sequence dimensions, it yields one importance score per neuron, which can be accumulated during calibration. section 5.2.1 section 5.5

Why use L2 for activation importance?

For [+5, -5, +5, -5], a simple sum cancels to zero, while the L2 norm is 10. It also gives [10, 0, 0, 0] a higher score than [5, 5, 0, 0] (about 7.07), emphasizing sharp activation peaks. section 5.2.1

Listing 5.4 Configuring hooks to capture MLP activation norms
_accumulated_act_norms = {} #1
 
def setup_mlp_hooks_for_importance(model, device):
    global _accumulated_act_norms
    _accumulated_act_norms.clear() #2
    gc.collect() #2
    torch.cuda.empty_cache() #2
    handles = []
 
    for idx, layer in enumerate(model.model.layers):
        intermediate_size = layer.mlp.down_proj.in_features
        _accumulated_act_norms[idx] = torch.zeros(
            intermediate_size,
            dtype=torch.float32,
            device='cpu' #3
        )
 
    def make_hook(layer_idx): #4
        def hook(module, input, output): #5
 
            X_d = input[0].detach() #6
 
            act_norms_L2 = torch.norm(
                X_d.to(torch.float32),
                p=2,
                dim=(0, 1)  #7
            )
            _accumulated_act_norms[layer_idx] +=
 ➥act_norms_L2.cpu() #8
 
       return hook
 
    for idx, layer in enumerate(model.model.layers):
        handle = layer.mlp.down_proj.register_forward_hook(
            make_hook(idx)
        ) #9
        handles.append(handle)
 
    print(f"✓ Registered {len(handles)} hooks on down_proj")
 
    return handles
 
def get_activation_norms():
    return {
        layer_idx: norms.clone()
        for layer_idx, norms in _accumulated_act_norms.items()
    }

The hook listing shows how to compute L2 activation norms at each MLP down-projection input and accumulate per-neuron scores. section 5.2.1

How Rearchitecting LLMs covers it

How deep it goes
Defines the square-root-of-squared-components calculation and reduces batch and sequence dimensions to per-neuron scores.
Examples
Compares alternating positive and negative activations with a single activation spike and evenly spread values.
What you can do after
Apply PyTorch activation norms to score MLP neurons by observed usage during calibration for width pruning.
How it connects
Connects activation norms to width pruning, forward hooks, and structural importance scores.

Also explained in

Compare how 7 books explain l2 norm →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.