liveBook · Manning

Attention sink

As explained in Build AI Drug Discovery Pipelines

An attention sink is a sequence position that attracts strong attention from many queries and acts as a shared reference point.

An attention sink is a sequence position that attracts strong attention from many query positions, acting as a shared reference point for global context. In protein language models, a vertical stripe in an attention heatmap reveals this pattern; position 0 is a common location. section 12.3.3

Attention patterns distinguish several roles among Transformer heads:

  • Local heads attend mainly to each position and its neighbors. section 12.3.3
  • Attention-sink heads emphasize one reference position across many queries. section 12.3.3
  • Other heads distribute attention broadly or connect distant residues that may interact structurally or co-evolve. section 12.3.3
Figure 12.11 Attention weights from each of the 4 heads in layer 4 (the final Transformer layer) reveal a diversity of learned patterns. Each heatmap shows query positions (Y-axis) attending to key positions (X-axis) for the sequence MKTAYIAKQRQISFVKSHFSRQLEERLGLI (~37 AAs). The viridis colormap indicates attention weight magnitude (darker purple = lower weight, brighter yellow-green = higher weight). Some heads learn to focus on local neighbors (tending towards the diagonal), while others attend to distant residues (further off-diagonal spots) or broad global context, demonstrating how the model divides the labor of understanding protein structure.

Inspect the four final-layer attention heatmaps to compare sink-like vertical stripes with local, distant, and diffuse attention patterns. section 12.3.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines the pattern and contrasts it with other head behaviors, but does not develop a separate mechanism explaining its emergence.
Examples
Uses final-layer heatmaps for a roughly 37-residue protein sequence to show vertical stripes, diagonal locality, and distant attention.
What you can do after
Evaluate attention-head heatmaps to recognize shared reference positions and distinguish them from local or long-range attention.
How it connects
Connects attention sinks with multi-head attention, Transformer encoders, protein language models, and long-range dependencies.

Also explained in

Compare how 2 books explain attention sink →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.