liveBook · Manning

Chapter 10: Exploring the transformer black box

What you will learn in chapter 10 of Rearchitecting LLMs

The introduction frames transformer activations as a way to inspect model behavior and briefly explains what activation heatmaps and the logit lens reveal.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction frames transformer activations as a way to inspect model behavior and briefly explains what activation heatmaps and the logit lens reveal.

Key terms: Transformer block, MLP module, Attention module, normalization layers, Activation heatmaps, neurons, token, depth, Logit lens, internal state

10.1 Capturing activations in transformers

This section shows how to locate submodule capture points in a transformer and use temporary hooks to collect activations from a prompt.

Key terms: hook point, submodule, Transformer block, gate_proj, up_proj, down_proj, self_attn, linear_attn, MLP, attention

10.2 Mapping neuron activity across the model

The section motivates heatmaps as a way to visualize large activation vectors and explains how their colors summarize neuron activity.

Key terms: activation, gate_proj, heatmaps, value bins, mean activation, color intensity

10.2.1 Two views of the same prompt

The section builds two heatmaps from the same prompt and activation capture point: one compares tokens within a layer, while the other shows how token-averaged activity changes across layers.

Key terms: BOS token, bin_neurons, bin_size, 99th percentile, attention sink, visualize_prompt_heatmap, token × neuron-bin matrix, layer_key, act_fn, visualize_prompt_layer_heatmap

10.2.2 The GLU mechanism in action

This section uses a three-panel activation visualization to show how the GLU combines its branches, what the resulting heatmaps reveal, and what they cannot reveal.

Key terms: GLU, act_fn, SiLU(gate_proj), up_proj, down_proj_input, pair-wise multiplication, visualize_glu_three_panel, get_prompt_activations, plot_heatmap_panel, layer 12

10.3 From noise to answer across the layers

The section uses a logit lens to read each transformer layer’s hidden state as a token prediction and show how a response emerges across layers.

Key terms: output_hidden_states=True, hidden_states, embedding output, hooks, logit lens, last token, final_norm, lm_head, logits, softmax

10.4 From paper to practice

The section relates the chapter’s activation and logit-lens tools to research alternatives, explaining what those alternatives offer and why the book uses lighter-weight methods.

Key terms: mechanistic interpretability, sparse autoencoders, activations, expanded layer, MLP expansion layers, data and compute, logit lens, hidden states, unembedding matrix, Pre-LayerNorm

10.5 Hands-on lab

10.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 9: Dynamic routing with Mixture of Experts Chapter 11: Precision pruning for bias →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.