liveBook · Manning

Capture point

As explained in Rearchitecting LLMs

A capture point is a specified model location at which intermediate activations are recorded during computation.

A capture point names a location in a transformer where intermediate activations are recorded. It identifies a module and whether to capture its input or output. section 10.1

Common capture points include:

  • MLP activation and projection outputs, the complete MLP output, and the input to down_proj. section 10.1
  • Attention output and input normalization output; attention module names vary across model families. section 10.1

Forward hooks capture module outputs; a forward pre-hook captures inputs. The down_proj_input point uses a pre-hook to record expanded MLP information before down_proj compresses it. Missing module names can cause a hook to be silently skipped. section 10.1

Table 10.2 Capture points with paths
Capture point
Path
Mechanism
Llama
Gemma
Qwen

act_fn
layer.mlp.act_fn
Forward hook
✓
✓
✓
gate_proj
layer.mlp.gate_proj
Forward hook
✓
✓
✓
up_proj
layer.mlp.up_proj
Forward hook
✓
✓
✓
down_proj_input
layer.mlp.down_proj
Forward pre-hook
✓
✓
✓
mlp_output
layer.mlp
Forward hook
✓
✓
✓
attention
layer.self_attn
Forward hook
✓
✓
✗
input_norm
layer.input_layernorm
Forward hook
✓
✓
✓

Compare capture-point paths and hook mechanisms across Llama, Gemma, and Qwen. section 10.1

Inspecting a prompt

Capture act_fn activations for “The cat sleeps” and visualize token-by-neuron responses as a heatmap. Neurons can be grouped into bins to keep the display manageable. section 10.2.1

How Rearchitecting LLMs covers it

How deep it goes
Distinguishes output hooks from input pre-hooks and explains expanded MLP capture, while warning that mismatched module names can silently skip hooks.
Examples
Shows MLP and attention capture locations, plus an act_fn heatmap for “The cat sleeps” with neurons binned.
What you can do after
Apply PyTorch hooks to record selected module inputs or outputs and visualize prompt activations across tokens or layers.
How it connects
Connects transformer layers and MLPs to attention modules, forward hooks, and activation heatmaps.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.