In Llama-family models, the automatically inserted BOS token can act as an attention sink: it accumulates unusually large activations that other tokens use as a reference, without carrying useful semantic information for the visualization. Exclude it before plotting token activations so it does not distort the heatmap. section 10.2.1
Listing 10.5 Token × neuron heatmap for a single layer
def visualize_prompt_heatmap(model, tokenizer, prompt, layer_key, bin_size=64,
cmap="RdBu_r", vmax_percentile=99.0):
layer_prefix = layer_key.split("_layer_")[0]
activations = get_prompt_activations(model, tokenizer, prompt, target_layers=[layer_prefix])
token_labels, start = get_token_labels(
tokenizer, prompt)
tensor = activations[layer_key][0][start:].float().numpy() #1
matrix = bin_neurons(tensor, bin_size) #2
fig, ax = plt.subplots(figsize=(14, max(4, len(token_labels) * 0.45 + 2.0)))
plot_heatmap_panel(ax, matrix, bin_size, cmap, vmax_percentile,
title=f"Activation Heatmap — {layer_key} | bin={bin_size}",
y_labels=token_labels) #A
ax.set_ylabel("Token position", fontsize=11)
plt.tight_layout()
plt.show()
See the heatmap workflow remove the BOS position before turning captured activations into a token-by-neuron matrix. section 10.2.1

