liveBook · Manning

Cross-attention

As explained in Build AI Drug Discovery Pipelines

Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values derived from another.

Cross-attention lets one representation select information from another. In an encoder–decoder model, the decoder uses its current state to attend to the encoder’s hidden states, rather than relying on a single vector to summarize the entire input. section 12.3.1

At each output step, the decoder state acts as a query. It is compared with encoder states acting as keys to produce attention weights. Those weights determine how much each encoder state, acting as a value, contributes to a weighted-sum context vector for that step. Different output steps can therefore draw on different input positions. section 12.3.1

Figure 12.8 The mechanics of cross-attention. A decoder state acts as a query (qj) and compares itself against all encoder states (keys, K). These comparisons generate attention weights (α), which are used to compute a weighted sum of the encoder states (values, V), creating a context vector (cj) specific to the current prediction step.

Follow how a decoder query produces weights over encoder states and a context vector for the current prediction. section 12.3.1

Distant residues in a protein

Suppose a protein has 500 amino acids, and the task is to predict whether cysteines at positions 45 and 312 form a disulfide bond. Instead of forcing the final encoder state to hold both contexts, cross-attention can give the decoder access to the hidden states at those positions and elsewhere in the sequence. section 12.3.1

A standard transformer uses cross-attention in its decoder to access the encoder’s sequence representations at each generation step. An encoder-only transformer instead keeps the encoder and discards the decoder; ESM-2 uses this approach to represent protein sequences. section 12.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains query-to-key comparisons, attention weights, and the weighted context vector for each decoding step.
Examples
Uses a 500-residue protein and a possible disulfide bond between cysteines at positions 45 and 312.
What you can do after
Understand how a decoder can draw on different input positions for each output.
How it connects
Connects encoder–decoder models, recurrent neural networks, and transformer architectures.

Also explained in

Compare how 8 books explain cross-attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.