In width pruning, the L2 norm measures activation intensity: square the components, sum them, and take the square root. Applied to [B, S, I] activations across batch and sequence dimensions, it yields one importance score per neuron, which can be accumulated during calibration. section 5.2.1 section 5.5
Why use L2 for activation importance?
For [+5, -5, +5, -5], a simple sum cancels to zero, while the L2 norm is 10. It also gives [10, 0, 0, 0] a higher score than [5, 5, 0, 0] (about 7.07), emphasizing sharp activation peaks. section 5.2.1
Listing 5.4 Configuring hooks to capture MLP activation norms
_accumulated_act_norms = {} #1
def setup_mlp_hooks_for_importance(model, device):
global _accumulated_act_norms
_accumulated_act_norms.clear() #2
gc.collect() #2
torch.cuda.empty_cache() #2
handles = []
for idx, layer in enumerate(model.model.layers):
intermediate_size = layer.mlp.down_proj.in_features
_accumulated_act_norms[idx] = torch.zeros(
intermediate_size,
dtype=torch.float32,
device='cpu' #3
)
def make_hook(layer_idx): #4
def hook(module, input, output): #5
X_d = input[0].detach() #6
act_norms_L2 = torch.norm(
X_d.to(torch.float32),
p=2,
dim=(0, 1) #7
)
_accumulated_act_norms[layer_idx] +=
➥act_norms_L2.cpu() #8
return hook
for idx, layer in enumerate(model.model.layers):
handle = layer.mlp.down_proj.register_forward_hook(
make_hook(idx)
) #9
handles.append(handle)
print(f"✓ Registered {len(handles)} hooks on down_proj")
return handles
def get_activation_norms():
return {
layer_idx: norms.clone()
for layer_idx, norms in _accumulated_act_norms.items()
}
The hook listing shows how to compute L2 activation norms at each MLP down-projection input and accumulate per-neuron scores. section 5.2.1






