liveBook · Manning

Register_forward_pre_hook

As explained in Rearchitecting LLMs

register_forward_pre_hook is a PyTorch method that registers a callback to run before a module's forward computation.

A PyTorch pre-hook runs before its module’s forward computation and receives the module’s input arguments. Use it to capture a tensor at a specific point in a model; unlike a forward hook, it captures the input rather than the output. section 8.3.1 section 10.1

A typical capture workflow:

  • Register a callback on each target module, often using a factory to associate the callback with a layer index. section 8.3.1 section 10.1
  • Read the input tensor from args[0] or inputs[0], then detach it before storing it. section 8.3.1 section 10.1
  • Keep the returned handles and remove them when capture is finished; otherwise, hooks continue running on later model executions. section 10.1

For attention-layer scoring, pre-hooks on input_layernorm and post_attention_layernorm capture hidden states before those modules execute. The latter input already includes the attention output added through the residual connection. section 8.3.1

Listing 10.2 Hook registration at capture points
def register_hooks(model, target_layers=None):
    activations = {}
    model._activations = activations
    handles = []
 
    def hook_fn(name):
        def hook(module, input, output):
            tensor = output[0] if isinstance(output, 
                                tuple) else output #A
            activations[name] = tensor.detach().cpu()
        return hook
 
    def pre_hook_fn(name):
        def hook(module, inputs):
            activations[name] = inputs[0].detach().cpu() #B
        return hook
 
    def capture(key): #C
        return target_layers is not None and key in target_layers
 
    for i, layer in enumerate(model.model.layers):
 
        if capture("act_fn") and hasattr(layer.mlp, "act_fn"): 
                handles.append(
                   layer.mlp.act_fn.register_forward_hook(
                       hook_fn(f"act_fn_layer_{i}"))
                ) #1
        if capture("gate_proj") and hasattr(layer.mlp, "gate_proj"): 
                handles.append(
                   layer.mlp.gate_proj.register_forward_hook(
                       hook_fn(f"gate_proj_layer_{i}"))
                ) #1
        if capture("down_proj_input") and hasattr(layer.mlp, "down_proj"):
                handles.append(
                   layer.mlp.down_proj.register_forward_pre_hook(
                       pre_hook_fn(f"down_proj_input_layer_{i}"))
                ) #1
 
    ....
 
    return handles

See the registration pattern for choosing capture points and saving pre-hook inputs. section 10.1

How Rearchitecting LLMs covers it

How deep it goes
Explains callback inputs, capture timing, detaching tensors, and handle cleanup without detailing hook internals.
Examples
Uses down_proj inputs for activation capture and input_layernorm/post_attention_layernorm tensors for attention scoring.
What it assumes
Requires identifying target modules and their input tensors within the model structure.
What you can do after
Build it from scratch by registering callbacks to save module inputs and removing their handles after capture.
How it connects
Connects forward hooks, activation capture, transformer layers, cosine similarity, and residual connections.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.