Multi-head attention combines several independently learned attention mechanisms, or heads, so a model can capture different relationships at the same time. In a protein sequence, one head might focus on nearby amino acid residues while another attends to residues farther away. section 12.3.2
In a transformer, each head has its own learned query, key, and value matrices. Each independently performs scaled dot-product attention over the sequence. The head outputs are concatenated and projected back to the model’s original dimension, combining their information for the next layer. section 12.3.2
Self-attention alone does not encode sequence order: shuffling the residues changes the output order but not the corresponding attention calculation. Positional encoding supplies position information alongside the residue embeddings. section 12.3.2 section 12.3.3
Heads in a protein language model
A protein language model adds positional encoding to amino acid embeddings, then passes them through a transformer encoder. With four sequences of length 50 and embedding dimension 128, the encoder maintains the shape (4, 50, 128); a language-modeling head then produces vocabulary predictions at each position. section 12.3.3
Figure 12.11 Attention weights from each of the 4 heads in layer 4 (the final Transformer layer) reveal a diversity of learned patterns. Each heatmap shows query positions (Y-axis) attending to key positions (X-axis) for the sequence MKTAYIAKQRQISFVKSHFSRQLEERLGLI (~37 AAs). The viridis colormap indicates attention weight magnitude (darker purple = lower weight, brighter yellow-green = higher weight). Some heads learn to focus on local neighbors (tending towards the diagonal), while others attend to distant residues (further off-diagonal spots) or broad global context, demonstrating how the model divides the labor of understanding protein structure.

The four attention maps show heads focusing on local neighbors, distant residues, or broader sequence context. Rows represent query positions and columns represent key positions. section 12.3.3
Multi-head attention also works on graphs. In a graph attention network, each head learns how to weight a node’s neighbors from their features. For a molecular graph, a head might learn to emphasize neighbors in the same functional group. Head outputs can be concatenated or averaged. section 11.5.4
self.mol_conv1 = GATConv(
num_features_mol, num_features_mol, heads=heads, concat=False)This GATConv layer uses the configured number of heads; concat=False averages their outputs. Setting heads=2 would learn two attention mechanisms in parallel. section 11.5.4
More expressive attention is not automatically better for a prediction task. In reported drug-target affinity benchmarks, a simpler graph convolutional network outperformed the graph attention alternative, possibly because extra expressiveness led to overfitting. section 11.5.4
























