liveBook · Manning

Heat map

As explained in Rearchitecting LLMs

A heatmap is a visualization that represents numerical values through variations in color across an arranged set of elements.

A heat map represents numerical values through color. For Transformer activations, it makes wide neuron vectors easier to inspect by grouping neurons into bins and coloring each bin by its mean activation. section 10.2

Two layouts show different views of activation:

  • Plot tokens against neuron bins to compare how a layer responds across a prompt. section 10.2.1
  • Average activations across tokens, then plot layers against neuron bins to follow changes through the model. section 10.2.1
  • For readable plots, bin neurons and use the 99th percentile of absolute values to set the color bound; discard the BOS token before plotting. section 10.2.1

Prompt activation view

For “The cat sleeps,” a token-by-neuron heat map uses one row per token and bins of 256 neurons per column. In the displayed layer, “cat” shows stronger responses across most bins than the other tokens. section 10.2.1

Figure 10.1 Token × neuron activation heatmap for act_fn_layer_12 on the prompt "The cat sleeps". Each row is a token, each column is a bin of 256 neurons averaged together. Activation intensity varies visibly across tokens: "cat" shows consistently stronger responses across most neuron bins, while "The" and "sleeps" remain more uniform. The BOS token has been excluded from the visualization.

View the token-by-neuron activation pattern for the prompt “The cat sleeps.” section 10.2.1

How Rearchitecting LLMs covers it

How deep it goes
Implements token-by-neuron and layer-by-neuron plots with binning, BOS exclusion, and percentile-based color scaling.
Examples
Shows “The cat sleeps” as token-by-neuron and layer-by-neuron views, with stronger responses for “cat” in the displayed layer.
What you can do after
Evaluate Transformer activation patterns by building heat maps across prompt tokens or model layers.
How it connects
Connects activation capture with Transformer layers, tokenization, neuron bins, and the GLU mechanism.

Also explained in

Compare how 10 books explain heat map →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.