Dynamic importance measures how strongly intermediate neurons activate on calibration inputs. In width pruning, activation norms complement weight-based structural importance, favoring neurons that are both structurally significant and actively used. section 5.2.1 section 5.5
Forward hooks capture activations at the input to down_proj, after gating and before dimensionality reduction. They compute L2 norms and accumulate results on the CPU to limit GPU memory use during offline calibration. section 5.2.1 section 5.5
Listing 5.4 Configuring hooks to capture MLP activation norms
_accumulated_act_norms = {} #1
def setup_mlp_hooks_for_importance(model, device):
global _accumulated_act_norms
_accumulated_act_norms.clear() #2
gc.collect() #2
torch.cuda.empty_cache() #2
handles = []
for idx, layer in enumerate(model.model.layers):
intermediate_size = layer.mlp.down_proj.in_features
_accumulated_act_norms[idx] = torch.zeros(
intermediate_size,
dtype=torch.float32,
device='cpu' #3
)
def make_hook(layer_idx): #4
def hook(module, input, output): #5
X_d = input[0].detach() #6
act_norms_L2 = torch.norm(
X_d.to(torch.float32),
p=2,
dim=(0, 1) #7
)
_accumulated_act_norms[layer_idx] +=
➥act_norms_L2.cpu() #8
return hook
for idx, layer in enumerate(model.model.layers):
handle = layer.mlp.down_proj.register_forward_hook(
make_hook(idx)
) #9
handles.append(handle)
print(f"✓ Registered {len(handles)} hooks on down_proj")
return handles
def get_activation_norms():
return {
layer_idx: norms.clone()
for layer_idx, norms in _accumulated_act_norms.items()
}
The listing shows how to initialize per-layer storage and configure hooks to accumulate MLP activation norms. section 5.2.1
Calibration data changes which neurons survive
Applying the same pruning percentage with WikiText versus SMS calibration preserves neurons suited to formal text versus conversational language. section 5.5
