Chapter 10: Exploring the transformer black box
The introduction frames transformer activations as a way to inspect model behavior and briefly explains what activation heatmaps and the logit lens reveal.
After this chapter you will be able to
- Inspectable Transformer components. Explain that a Transformer block can be examined as a whole or through specific components, including its MLP, attention, and normalization modules.
- Hierarchical activation capture points. Explain why inspecting the model's module hierarchy comes before attaching hooks, and describe how both whole modules and their internal projections can be captured independently.
- Input versus output capture. Distinguish forward hooks from forward pre-hooks and explain why down_proj_input captures the input to down_proj.
- Activation capture and cleanup. Trace how requested hooks collect a prompt's activations and explain why their handles must be removed after the forward pass, including on failure.
- Activation tensors and heatmaps. Explain why large activation vectors are represented as heatmaps and how a heatmap encodes their values, using the section’s four-token example.
- Single-layer token responses. Trace how the single-layer heatmap becomes a token × neuron-bin matrix, and describe what the example reveals about the prompt's tokens.
- Token-averaged depth changes. Explain how averaging tokens produces a layer × neuron-bin view, what information that averaging loses, and where the example shows stronger activity.
- GLU branch interaction and filtering. Explain how the gate and up-projection outputs combine to form the input to down_proj, and describe the filtering effect shown in the layer 14 visualization.
- Layer states as token predictions. Trace how the logit lens turns each block’s last-token hidden state into a ranked list of likely next tokens.
- Predictions evolving across layers. Describe the progression shown by the Barcelona prompt and explain what it suggests about reading intermediate layers.
- Polysemantic neurons and sparse autoencoders. Explain why a single neuron’s activation can be hard to interpret and how a sparse autoencoder can produce more topic-specific dimensions for studying or influencing a model.
- Logit lens predictions and limits. Describe what the logit lens reveals about predictions across layers and state the model-architecture limitation identified here.
- Tuned lens accuracy trade-off. Distinguish the tuned lens from the logit lens and explain why the book uses the latter under its resource constraints.
What it builds on
- General behaviour of a Transformer model (chapter 3) · The earlier account of Transformer model structure.
- Using PyTorch hooks (chapter 4) · Introduced hooks for observing entire transformer blocks.
- The evolution of the MLP: from simple expansion to Gated Linear Units (GLU) (chapter 3) · The GLU components and their role in the modern transformer MLP
Section by section
Introduction
The introduction frames transformer activations as a way to inspect model behavior and briefly explains what activation heatmaps and the logit lens reveal.
- Inspectable Transformer components: Explain that a Transformer block can be examined as a whole or through specific components, including its MLP, attention, and normalization modules.
Why it matters: It gives you a choice of where to look when examining activity inside a block. - Activation heatmap views: Describe what activation heatmaps show across tokens and model depth.
Why it matters: Heatmaps give a view of how neuron responses vary by token and depth. - Logit lens projection: Explain, at a high level, how the logit lens makes the developing output visible across layers.
Why it matters: It offers a layer-by-layer view of how the output takes shape.
Key terms: Transformer block, MLP module, Attention module, normalization layers, Activation heatmaps, neurons, token, depth, Logit lens, internal state
10.1 Capturing activations in transformers
This section shows how to locate submodule capture points in a transformer and use temporary hooks to collect activations from a prompt.
- Hierarchical activation capture points: Explain why inspecting the model's module hierarchy comes before attaching hooks, and describe how both whole modules and their internal projections can be captured independently.
Why it matters: Finding the actual submodules lets you choose the activity you want to inspect rather than capturing only a whole block. - Architecture-specific module names: Explain why hook code must use the module names exposed by the particular model, using the Qwen3.5 attention example.
Why it matters: A missing capture may reflect a module-name mismatch rather than an absence of activity. - Input versus output capture: Distinguish forward hooks from forward pre-hooks and explain why down_proj_input captures the input to down_proj.
Why it matters: The side of the layer you capture determines whether you see the expanded representation or its compressed result. - Activation capture and cleanup: Trace how requested hooks collect a prompt's activations and explain why their handles must be removed after the forward pass, including on failure.
Why it matters: Removing hooks prevents later model runs from silently continuing to capture activations.
Key terms: hook point, submodule, Transformer block, gate_proj, up_proj, down_proj, self_attn, linear_attn, MLP, attention
10.2 Mapping neuron activity across the model
The section motivates heatmaps as a way to visualize large activation vectors and explains how their colors summarize neuron activity.
- Activation tensors and heatmaps: Explain why large activation vectors are represented as heatmaps and how a heatmap encodes their values, using the section’s four-token example.
Why it matters: This explains what the colors in the upcoming activation charts represent.
Key terms: activation, gate_proj, heatmaps, value bins, mean activation, color intensity
10.2.1 Two views of the same prompt
The section builds two heatmaps from the same prompt and activation capture point: one compares tokens within a layer, while the other shows how token-averaged activity changes across layers.
- Readable activation heatmaps: Explain why the visualizations exclude the BOS token, average neurons into bins, and use a percentile rather than the absolute maximum for the color scale.
Why it matters: A readable color scale makes comparisons in both views meaningful. - Single-layer token responses: Trace how the single-layer heatmap becomes a token × neuron-bin matrix, and describe what the example reveals about the prompt's tokens.
Why it matters: This view shows which prompt tokens elicit different responses at one point in the model. - Token-averaged depth changes: Explain how averaging tokens produces a layer × neuron-bin view, what information that averaging loses, and where the example shows stronger activity.
Why it matters: This complementary view locates changes across the network that a single-layer token plot cannot show.
Key terms: BOS token, bin_neurons, bin_size, 99th percentile, attention sink, visualize_prompt_heatmap, token × neuron-bin matrix, layer_key, act_fn, visualize_prompt_layer_heatmap
10.2.2 The GLU mechanism in action
This section uses a three-panel activation visualization to show how the GLU combines its branches, what the resulting heatmaps reveal, and what they cannot reveal.
- GLU branch interaction and filtering: Explain how the gate and up-projection outputs combine to form the input to down_proj, and describe the filtering effect shown in the layer 14 visualization.
Why it matters: The combined output shows what the GLU passes onward, which a gate-only heatmap cannot show. - Three-panel activation comparison: Describe how the visualization puts three GLU capture points side by side for a comparable view of one prompt and layer.
Why it matters: A shared prompt, layer, and rendering approach make differences between the GLU components easier to interpret. - Layerwise token activity pattern: Describe how the act_fn activity for “cat” differs between the layer 12 and layer 14 heatmaps.
Why it matters: The comparison gives a concrete example of how the visible activation pattern changes across layers. - Limits of activation heatmaps: Distinguish what neuron-activation visualizations show from what they do not reveal about the model’s state.
Why it matters: It keeps an activation heatmap from being mistaken for a direct account of the model’s meaning or internal reasoning.
Key terms: GLU, act_fn, SiLU(gate_proj), up_proj, down_proj_input, pair-wise multiplication, visualize_glu_three_panel, get_prompt_activations, plot_heatmap_panel, layer 12
10.3 From noise to answer across the layers
The section uses a logit lens to read each transformer layer’s hidden state as a token prediction and show how a response emerges across layers.
- Hidden states for the lens: Explain how the section obtains hidden states for every layer and why it sets aside the embedding output.
Why it matters: It identifies the data the logit lens reads and prevents the embedding output from being mistaken for a block output. - Layer states as token predictions: Trace how the logit lens turns each block’s last-token hidden state into a ranked list of likely next tokens.
Why it matters: This explains what the per-layer token table measures—and why its entries are readings of intermediate states, not ordinary final predictions. - Predictions evolving across layers: Describe the progression shown by the Barcelona prompt and explain what it suggests about reading intermediate layers.
Why it matters: It shows why a plausible token at an earlier layer need not be the token favored after all layers have run.
Key terms: output_hidden_states=True, hidden_states, embedding output, hooks, logit lens, last token, final_norm, lm_head, logits, softmax
10.4 From paper to practice
The section relates the chapter’s activation and logit-lens tools to research alternatives, explaining what those alternatives offer and why the book uses lighter-weight methods.
- Polysemantic neurons and sparse autoencoders: Explain why a single neuron’s activation can be hard to interpret and how a sparse autoencoder can produce more topic-specific dimensions for studying or influencing a model.
Why it matters: It explains why observing or changing one original neuron does not necessarily isolate one concept. - Resource-conscious MLP activation study: Justify the book’s choice to study MLP expansion-layer activations rather than train sparse autoencoders.
Why it matters: It makes clear that the simpler activation study is a resource-conscious compromise, not the same thing as training sparse autoencoders. - Logit lens predictions and limits: Describe what the logit lens reveals about predictions across layers and state the model-architecture limitation identified here.
Why it matters: It helps the reader interpret layer-by-layer prediction views without assuming they are reliable for every architecture. - Tuned lens accuracy trade-off: Distinguish the tuned lens from the logit lens and explain why the book uses the latter under its resource constraints.
Why it matters: It explains the choice between a more accurate trained lens and an immediately usable, lower-cost one.
Key terms: mechanistic interpretability, sparse autoencoders, activations, expanded layer, MLP expansion layers, data and compute, logit lens, hidden states, unembedding matrix, Pre-LayerNorm
10.5 Hands-on lab
10.6 Summary
Where this is used later
- (chapter intro) (chapter 11) · Capturing activations in transformers.
- Comparing activations between prompts (chapter 11) · Compares activations collected from different prompts.
- Anthropic, the starting point (chapter 11) · Observation of neuron activity within the model
- Where the differences show up (chapter 11) · The GLU mechanism within the MLP.
- Intervening on the top-k neurons (chapter 11) · The GLU branches and their roles in neuron activity.
- Tracing the effect across layers (chapter 11) · The earlier introduction of the logit lens for inspecting predictions across layers.
Concepts explained in this chapter
- logit lensThe logit lens is an interpretability technique that projects intermediate hidden states through a model's final normalization and prediction head to inspect layer-wise predictions.
- capture pointA capture point is a specified model location at which intermediate activations are recorded during computation.
- hookA model hook is a callback attached to a neural-network component that observes or modifies computation at a specified point during execution.
- heat mapA heatmap is a visualization that represents numerical values through variations in color across an arranged set of elements.
- multilayer perceptronA multilayer perceptron is a feed-forward neural network composed of successive learned linear transformations and nonlinear activations.
- pre-hookA pre-hook is a callback that receives a module's input arguments before the module executes.
- attentionAttention is a neural-network mechanism that computes input-dependent weights to prioritize information during processing.
- activation captureActivation capture records intermediate neural signals produced during model execution.
- down projection layerA down projection is a linear transformation that reduces an intermediate representation back to the model's hidden dimension.
- gate projectionA gate projection produces the signal that controls another intermediate signal in a gated feed-forward block.
- register_forward_hookregister_forward_hook is a PyTorch method that registers a callback to run after a module's forward computation.
- register_forward_pre_hookregister_forward_pre_hook is a PyTorch method that registers a callback to run before a module's forward computation.
- up projectionAn up projection is a feed-forward linear transformation that expands a hidden representation into an intermediate dimension.
- attention sinkAn attention sink is a sequence position that accumulates unusually large activations and serves as a reference for other tokens without carrying corresponding semantic information.
- beginning-of-sequence tokenA BOS token is a beginning-of-sequence marker automatically inserted at the start of a tokenized sequence.
- percentileA percentile is a value below which a specified percentage of observations fall in an ordered distribution.
- activationAn activation is a value produced by a neural-model unit during computation.
- gated linear unitA gated linear unit is a neural-network structure that modulates one linear transformation with a learned gate derived from another transformation.
- sparse autoencoderA sparse autoencoder is an autoencoder that reconstructs neural activations through a wider intermediate representation constrained to activate sparsely.
- hidden stateA hidden state is an internal vector representation produced by a neural model layer for the information processed up to that layer.
- language modeling headA language modeling head maps model representations to scores over a language model's vocabulary.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
