liveBook · Manning

Activation norm

As explained in Rearchitecting LLMs

An activation norm is a numerical magnitude of a neural activation signal, used to quantify how strongly a unit responds to input data.

An activation norm measures the magnitude of a neural activation signal. In data-driven width pruning, it quantifies how strongly each MLP neuron responds to calibration inputs and supplies the dynamic part of a hybrid importance score. section 5.2 section 5.2.1

Capture the signal at the input of down_proj, after gating but before the projection mixes and contracts the intermediate neurons. For Llama-3.2-1B, this projection contracts 8192 intermediate neurons to 2048 hidden dimensions. section 5.2 section 5.5

Collect per-neuron norms during calibration:

  • Register a PyTorch forward hook on each MLP down_proj. Allocate a float32 CPU accumulator sized to that layer’s intermediate dimension. section 5.2.1
  • In each hook, detach the input, convert it to float32, and calculate its L2 norm across batch and sequence dimensions. Add the resulting per-neuron vector to the CPU accumulator. section 5.2.1
  • Run calibration batches in evaluation mode with gradients disabled, then remove the hooks. Clear GPU cache periodically if memory is constrained, accepting a possible speed cost. section 5.2.3

The accumulated statistic is a sum of per-batch L2 norms, not a single L2 norm over the entire dataset. Keeping only these vectors on CPU avoids retaining raw activation tensors on the GPU. section 5.2.1

Listing 5.4 Configuring hooks to capture MLP activation norms
_accumulated_act_norms = {} #1
 
def setup_mlp_hooks_for_importance(model, device):
    global _accumulated_act_norms
    _accumulated_act_norms.clear() #2
    gc.collect() #2
    torch.cuda.empty_cache() #2
    handles = []
 
    for idx, layer in enumerate(model.model.layers):
        intermediate_size = layer.mlp.down_proj.in_features
        _accumulated_act_norms[idx] = torch.zeros(
            intermediate_size,
            dtype=torch.float32,
            device='cpu' #3
        )
 
    def make_hook(layer_idx): #4
        def hook(module, input, output): #5
 
            X_d = input[0].detach() #6
 
            act_norms_L2 = torch.norm(
                X_d.to(torch.float32),
                p=2,
                dim=(0, 1)  #7
            )
            _accumulated_act_norms[layer_idx] +=
 ➥act_norms_L2.cpu() #8
 
       return hook
 
    for idx, layer in enumerate(model.model.layers):
        handle = layer.mlp.down_proj.register_forward_hook(
            make_hook(idx)
        ) #9
        handles.append(handle)
 
    print(f"✓ Registered {len(handles)} hooks on down_proj")
 
    return handles
 
def get_activation_norms():
    return {
        layer_idx: norms.clone()
        for layer_idx, norms in _accumulated_act_norms.items()
    }

The hook listing shows the reduction dimensions, detachment, and per-layer CPU accumulation. section 5.2.1

Combine each activation norm with a structural score derived from gate_proj, up_proj, and down_proj weights. Select the highest hybrid scores, retain at least one neuron, and rebuild the narrower projections using the selected indices. Validate that every layer has activation data and update the model configuration after surgery. section 5.2 section 5.2.2 section 5.2.4

Calibration data determine which activity is observed. A small domain-specific dataset may miss structurally important neurons; multiplying by structural importance does not guarantee preservation of a neuron whose measured activation is zero. section 5.2 section 5.2.4

Two calibration domains

Calibrate the same base model separately on WikiText-2 and SMS Spam, then apply the same 20% width pruning using each norm dictionary. The resulting models have the same parameter count but can retain different neurons. Each reported model has 1,074,792,448 parameters, down 13.03% from the 1,235,814,400-parameter baseline. section 5.2.3 section 5.2.4

How Rearchitecting LLMs covers it

How deep it goes
Details per-layer hook capture, batchwise L2 accumulation, hybrid scoring, and projection reconstruction with safety checks.
Examples
Compares WikiText-2 and SMS Spam calibration on Llama-3.2-1B with 20% width pruning and reports parameter counts.
What it assumes
Assumes prepared calibration datasets and an earlier static pruning workflow rather than reconstructing either from zero.
What you can do after
Apply it via PyTorch hooks to collect per-neuron calibration statistics and use hybrid scores to prune MLP projections.
How it connects
Connects activation norms to L2 norms, calibration, GLU gating, weight-based importance, and width pruning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.