liveBook · Manning

Hook

As explained in Rearchitecting LLMs

A model hook is a callback attached to a neural-network component that observes or modifies computation at a specified point during execution.

A PyTorch hook attaches a callback to a model module so it runs automatically during execution, without changing the model’s code. Hooks can capture activations for debugging, interpretability, or data-driven pruning. section 4.3.1 section 10.1

A forward hook registered with register_forward_hook receives the module, its input tuple, and its output after the forward call. A forward pre-hook captures inputs before the module runs. Choose the point according to the tensor needed: for example, the input to an MLP’s down_proj contains the expanded representation before compression. section 4.3.1 section 10.1

Inspect a decoder block

Attach a forward hook to model.model.layers[0], tokenize a prompt, and run a forward pass with gradients disabled. The callback can print the module class and input and output shapes. In one Qwen3-0.6B run, both shapes are [1, 8, 1024]: one sequence, eight tokens, and 1024 values per token. Remove the returned handle when finished. section 4.3.1

first_layer = model.model.layers[0]
hook_handle = first_layer.register_forward_hook(print_shape_hook)

Registers the shape callback on the first decoder layer and retains its removal handle. section 4.3.1

A calibration workflow can use hooks to select blocks for depth pruning:

  • Register one hook per block. A factory binds each layer index so captured inputs and outputs can be stored under the correct key; detach tensors and handle tuple outputs where necessary. section 4.3.3
  • Run calibration batches without gradients, then compare each block’s input and output with cosine similarity. Exclude padding with the attention mask and check for missing activations, empty tensors, and non-finite results. section 4.3.3
  • Aggregate scores across the task data and use them to identify pruning candidates. Importance depends on the calibration data and task, not only on model structure. section 4.3 section 4.3.3
Listing 4.5 Configuring hooks to capture activations.
def setup_layer_hooks(model):
 
    num_layers = len(model.model.layers)
    layer_inputs = {} #A
    layer_outputs = {} #A
    hooks = []
 
   def create_hook(layer_idx): #B
        def hook(module, input, output): #C
            input_tensor = input[0] if isinstance(input, tuple) else input #D
            layer_inputs[layer_idx] = 
➥input_tensor.detach() #E

            output_tensor = output[0] if isinstance(output, tuple) else output #D
            layer_outputs[layer_idx] = 
➥output_tensor.detach() #E
        return hook

    for i, layer in enumerate(model.model.layers):
        hooks.append(
            layer.register_forward_hook(create_hook(i))
        ) #F

    return hooks, layer_inputs, layer_outputs, num_layers

Shows per-layer callback creation, detached activation capture, and handle collection. section 4.3.3

For width pruning, hooks on MLP down_proj inputs can calculate activation L2 norms during calibration and accumulate them on the CPU. These dynamic measurements combine with normalized weight ranges to score neurons. Computing scores inside a hook can reduce activation storage, but sacrifices the ability to inspect raw tensors afterward. section 5.2.3 section 5.2 section 4.5

Inspect the actual module hierarchy before choosing hook points. Names vary across model families: attention may be self_attn or linear_attn, so a name-based capture can silently miss it. Verify that expected activations were captured. Remove every handle after use; a try/finally cleanup also handles failed forward passes. section 10.1 section 10.5

Listing 10.3 Hook cleanup and activation capture
def remove_hooks(handles):
    for handle in handles:
        handle.remove()
 
 
def get_prompt_activations(model, tokenizer, prompt, target_layers=None):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    handles = register_hooks(model, 
        target_layers=target_layers) #A
    Try: #1
        with torch.no_grad():
            model(**inputs)
        return {k: v.clone() for k, v 
                in model._activations.items()} #B
    finally:
        remove_hooks(handles)
        if hasattr(model, "_activations"):
            del model._activations

Shows prompt capture with hook removal and activation cleanup in a finally block. section 10.1

How Rearchitecting LLMs covers it

How deep it goes
Walks through callback signatures, registration, activation collection, calibration, validation, and cleanup, including an exercise to reduce storage.
Examples
Uses Qwen3 decoder shape inspection, WinoGrande block calibration, and MLP down-projection activation capture.
What you can do after
Applies hooks via PyTorch to capture and analyze activations, validate capture points, and remove callbacks after inference.
How it connects
Connects hooks to cosine similarity, depth pruning, width pruning, activation norms, and transformer interpretability.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.