liveBook · Manning

Attention sink

As explained in Rearchitecting LLMs

An attention sink is a sequence position that accumulates unusually large activations and serves as a reference for other tokens without carrying corresponding semantic information.

In Llama-family models, the automatically inserted BOS token can act as an attention sink: it accumulates unusually large activations that other tokens use as a reference, without carrying useful semantic information for the visualization. Exclude it before plotting token activations so it does not distort the heatmap. section 10.2.1

Listing 10.5 Token × neuron heatmap for a single layer
def visualize_prompt_heatmap(model, tokenizer, prompt, layer_key, bin_size=64,
                              cmap="RdBu_r", vmax_percentile=99.0):
    layer_prefix = layer_key.split("_layer_")[0]
    activations = get_prompt_activations(model, tokenizer, prompt, target_layers=[layer_prefix])
 
    token_labels, start = get_token_labels(
           tokenizer, prompt) 
    tensor = activations[layer_key][0][start:].float().numpy() #1
    matrix = bin_neurons(tensor, bin_size) #2
 
    fig, ax = plt.subplots(figsize=(14, max(4, len(token_labels) * 0.45 + 2.0)))
    plot_heatmap_panel(ax, matrix, bin_size, cmap, vmax_percentile,
                        title=f"Activation Heatmap — {layer_key} | bin={bin_size}",
                        y_labels=token_labels) #A
 
    ax.set_ylabel("Token position", fontsize=11)
    plt.tight_layout()
    plt.show()

See the heatmap workflow remove the BOS position before turning captured activations into a token-by-neuron matrix. section 10.2.1

How Rearchitecting LLMs covers it

How deep it goes
Identifies the BOS position’s high-activation reference role and explains its exclusion from heatmaps.
Examples
Uses the BOS token in a Llama prompt heatmap and excludes it before visualization.
What you can do after
Evaluate activation heatmaps without allowing BOS activations to distort their color scale.
How it connects
Connects BOS tokens with transformer activations and activation heatmaps.

Also explained in

Compare how 2 books explain attention sink →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.