Activation capture records intermediate neural signals during a model’s forward pass. Hooks make it possible to inspect individual transformer submodules or collect activation statistics for data-driven neuron selection. section 5.2.3 section 10.1
Choose the capture point according to the signal needed. A forward hook records a module’s output; a forward pre-hook records its input. For neuron-level MLP analysis, capture the input of down_proj: it contains the gated, expanded signal before down_proj mixes and compresses it. In Llama-3.2-1B, down_proj maps 8,192 features to 2,048, so its output no longer preserves the same per-neuron traceability. section 5.2 section 10.1
A practical capture workflow:
- Inspect the model hierarchy and select submodules and input or output hook points. Module names differ across architectures; a hook targeting self_attn can silently miss Qwen3.5’s linear_attn. section 10.1
- Register hooks for the selected layers. For prompt-level inspection, detach captured tensors and move them to CPU; handle tuple outputs by selecting the desired tensor. section 10.1
- Run the model without gradients. For calibration, use evaluation mode and process batches on the model’s device; hooks can compute L2 norms at each down_proj input and accumulate them on CPU. section 5.2.3
- Remove every hook after capture. A finally block ensures cleanup even if the forward pass fails. On a memory-constrained GPU, periodic cache clearing can reduce VRAM pressure at a speed cost. section 10.1 section 5.2.3
Listing 10.3 Hook cleanup and activation capture
def remove_hooks(handles):
for handle in handles:
handle.remove()
def get_prompt_activations(model, tokenizer, prompt, target_layers=None):
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
handles = register_hooks(model,
target_layers=target_layers) #A
Try: #1
with torch.no_grad():
model(**inputs)
return {k: v.clone() for k, v
in model._activations.items()} #B
finally:
remove_hooks(handles)
if hasattr(model, "_activations"):
del model._activations
The prompt-capture workflow registers hooks, runs without gradients, copies the results, and guarantees cleanup. section 10.1
Calibration for width pruning
Run calibration batches from WikiText-2 or SMS Spam through the model while hooks collect per-layer down_proj input norms. Combine the dynamic activation signal with normalized weight-range scores from gate_proj, up_proj, and down_proj. Multiplying the structural and activation scores lowers the importance of a neuron with large weights but near-zero activation, making it a pruning candidate. section 5.2.3 section 5.2
Figure 5.5 Hybrid importance calculation in GLU architecture. The static component combines weight ranges from gate_proj, up_proj, and down_proj. The dynamic component captures activation norms at the input of down_proj, where gated information flows. Both components multiply to produce the final importance score.
The diagram locates capture before down_proj and shows how activation norms combine with weight-based scores. section 5.2
