liveBook · Manning

Multi-head attention

As explained in Build AI Drug Discovery Pipelines

Multi-head attention is an attention mechanism that combines multiple independently parameterized attention operations to capture different relationships in a sequence.

Multi-head attention combines several independently learned attention mechanisms, or heads, so a model can capture different relationships at the same time. In a protein sequence, one head might focus on nearby amino acid residues while another attends to residues farther away. section 12.3.2

In a transformer, each head has its own learned query, key, and value matrices. Each independently performs scaled dot-product attention over the sequence. The head outputs are concatenated and projected back to the model’s original dimension, combining their information for the next layer. section 12.3.2

Self-attention alone does not encode sequence order: shuffling the residues changes the output order but not the corresponding attention calculation. Positional encoding supplies position information alongside the residue embeddings. section 12.3.2 section 12.3.3

Heads in a protein language model

A protein language model adds positional encoding to amino acid embeddings, then passes them through a transformer encoder. With four sequences of length 50 and embedding dimension 128, the encoder maintains the shape (4, 50, 128); a language-modeling head then produces vocabulary predictions at each position. section 12.3.3

Figure 12.11 Attention weights from each of the 4 heads in layer 4 (the final Transformer layer) reveal a diversity of learned patterns. Each heatmap shows query positions (Y-axis) attending to key positions (X-axis) for the sequence MKTAYIAKQRQISFVKSHFSRQLEERLGLI (~37 AAs). The viridis colormap indicates attention weight magnitude (darker purple = lower weight, brighter yellow-green = higher weight). Some heads learn to focus on local neighbors (tending towards the diagonal), while others attend to distant residues (further off-diagonal spots) or broad global context, demonstrating how the model divides the labor of understanding protein structure.

The four attention maps show heads focusing on local neighbors, distant residues, or broader sequence context. Rows represent query positions and columns represent key positions. section 12.3.3

Multi-head attention also works on graphs. In a graph attention network, each head learns how to weight a node’s neighbors from their features. For a molecular graph, a head might learn to emphasize neighbors in the same functional group. Head outputs can be concatenated or averaged. section 11.5.4

self.mol_conv1 = GATConv(
    num_features_mol, num_features_mol, heads=heads, concat=False)

This GATConv layer uses the configured number of heads; concat=False averages their outputs. Setting heads=2 would learn two attention mechanisms in parallel. section 11.5.4

More expressive attention is not automatically better for a prediction task. In reported drug-target affinity benchmarks, a simpler graph convolutional network outperformed the graph attention alternative, possibly because extra expressiveness led to overfitting. section 11.5.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains independent attention projections and output fusion, and demonstrates graph-layer configuration while deferring the full protein encoder implementation to accompanying resources.
Examples
Illustrates heads with protein attention heatmaps and a two-head molecular GATConv layer.
What you can do after
Apply it via GATConv by configuring the number of heads and choosing whether to concatenate or average their outputs.
How it connects
Connects attention to transformer encoders, positional encoding, graph attention networks, and graph convolutional networks.

Also explained in

Compare how 25 books explain multi-head attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.