liveBook · Manning

Attention

As explained in Build AI Drug Discovery Pipelines

An attention mechanism computes input-dependent weights that determine how strongly elements of a sequence or representation influence one another.

Attention computes input-dependent weights that determine how strongly different pieces of information contribute to a representation. In protein modeling, it can bring information from distant amino acids together: residues far apart in a sequence may interact when the protein folds. section 12.3 section 12.1.1

In cross-attention, a decoder uses its current state as a query to compare against encoder states, which serve as keys. The resulting weights combine the corresponding values into a context vector for that prediction step. This gives the decoder access to all encoder states rather than relying on a single compressed summary. section 12.3.1

Figure 12.8 The mechanics of cross-attention. A decoder state acts as a query (qj) and compares itself against all encoder states (keys, K). These comparisons generate attention weights (α), which are used to compute a weighted sum of the encoder states (values, V), creating a context vector (cj) specific to the current prediction step.

Follow how a decoder query produces weights over encoder states and a context vector. section 12.3.1

In self-attention, queries, keys, and values come from the same sequence representation. Each residue can attend to every position, including itself. An encoder-only transformer can update all positions in parallel, allowing distant residues to influence one another without specifying their interactions in advance. section 12.3.2 section 12.3

Predicting a masked amino acid

A protein language model can replace the K in MKTAYIAKQRQISF with a mask and learn to predict it from the surrounding sequence. Self-attention contextualizes residue embeddings for this task. Evaluate masked-token accuracy or pseudo-perplexity on held-out sequences, and check for near-duplicates leaking across the split. section 12.3.3

Multi-head attention learns several attention patterns in parallel. In a protein model, different heads may focus on nearby residues, distant positions, or broader context. These patterns can be inspected as heatmaps of query positions against key positions. section 11.5.4 section 12.3.3

Figure 12.11 Attention weights from each of the 4 heads in layer 4 (the final Transformer layer) reveal a diversity of learned patterns. Each heatmap shows query positions (Y-axis) attending to key positions (X-axis) for the sequence MKTAYIAKQRQISFVKSHFSRQLEERLGLI (~37 AAs). The viridis colormap indicates attention weight magnitude (darker purple = lower weight, brighter yellow-green = higher weight). Some heads learn to focus on local neighbors (tending towards the diagonal), while others attend to distant residues (further off-diagonal spots) or broad global context, demonstrating how the model divides the labor of understanding protein structure.

Compare local and distant attention patterns across four heads. section 12.3.3

Attention can also operate over a molecular graph, where atoms are nodes and bonds connect neighbors. A graph attention network learns edge weights from node features rather than using fixed degree normalization. Unlike global sequence attention, it aggregates information over predefined graph neighbors. section 11.5.4 section 12.3

self.mol_conv1 = GATConv(
    num_features_mol, num_features_mol, heads=heads, concat=False)

See a GATConv layer with parallel heads; concat=False averages their outputs. section 11.5.4

More elaborate attention is not automatically better. In reported drug-target affinity benchmarks, a simpler graph convolutional network performed best among the compared architectures, so attention-based models should be evaluated against appropriate baselines. section 11.5.4

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains query-key-value aggregation, self-attention and multi-head attention, then demonstrates graph layers and a small protein language model.
Examples
Illustrates attention with distant cysteines, masked amino acid prediction and heatmaps of protein-sequence attention heads.
What you can do after
Apply attention through GATConv and train a small protein language model, evaluating masked-token predictions on held-out sequences.
How it connects
Connects attention to graph neural networks, transformers, protein structure prediction and masked language modeling.

Also explained in

Compare how 40 books explain attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.