Attention sink
An attention sink is a sequence position that attracts attention from many queries and serves as a stable shared target during attention computation.
Rearchitecting LLMs
An attention sink is a sequence position that accumulates unusually large activations and serves as a reference for other tokens without carrying corresponding semantic information.
- How it is used
- Removes the automatically inserted BOS position before plotting activations, preventing its unusually large values from distorting the heatmap.
- How deep it goes
- Identifies the BOS position’s high-activation reference role and explains its exclusion from heatmaps.
- Examples
- Uses the BOS token in a Llama prompt heatmap and excludes it before visualization.
- What you can do after
- Evaluate activation heatmaps without allowing BOS activations to distort their color scale.
Build AI Drug Discovery Pipelines
An attention sink is a sequence position that attracts strong attention from many queries and acts as a shared reference point.
- How it is used
- Interprets attention heatmaps from a protein language model to identify positions providing shared global context.
- How deep it goes
- Defines the pattern and contrasts it with other head behaviors, but does not develop a separate mechanism explaining its emergence.
- Examples
- Uses final-layer heatmaps for a roughly 37-residue protein sequence to show vertical stripes, diagonal locality, and distant attention.
- What you can do after
- Evaluate attention-head heatmaps to recognize shared reference positions and distinguish them from local or long-range attention.
Read how Build AI Drug Discovery Pipelines explains attention sink →
Also covered in
Related concepts
- beginning-of-sequence tokenused by · A beginning-of-sequence token is a special token marking the start of a tokenized sequence, often signaling a generative decoder to begin output.
- FlashAttentionused by · FlashAttention is an exact, I/O-aware attention algorithm that tiles computation to reduce memory traffic without materializing the full attention matrix in high-bandwidth memory.
Questions about attention sink
What is attention sink?
An attention sink is a sequence position that attracts attention from many queries and serves as a stable shared target during attention computation.
Which Manning books explain attention sink?
Rearchitecting LLMs and Build AI Drug Discovery Pipelines.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

