liveBook · Manning

Attention weight

As explained in Build AI Drug Discovery Pipelines

An attention weight is a normalized numerical coefficient indicating how strongly an attention operation incorporates an associated value.

An attention weight is a normalized numerical coefficient showing how strongly an attention operation incorporates an associated value. In cross-attention, the decoder computes one weight for each encoder hidden state, based on its relevance to the current decoding step. section 12.3.1

Cross-attention uses the weights in a query-key-value workflow:

  • The decoder’s current hidden state supplies the query. section 12.3.1
  • Encoder hidden states supply the keys and values. section 12.3.1
  • Comparisons produce attention weights, which determine each hidden state’s contribution to the weighted-sum context vector. section 12.3.1

Protein structure prediction

When predicting structure at output position j, the decoder may assign higher weights to nearby sequence positions or to distant positions that interact with j in three-dimensional space. section 12.3.1

Figure 12.8 The mechanics of cross-attention. A decoder state acts as a query (qj) and compares itself against all encoder states (keys, K). These comparisons generate attention weights (α), which are used to compute a weighted sum of the encoder states (values, V), creating a context vector (cj) specific to the current prediction step.

Shows how decoder queries compare with encoder keys to generate weights and a position-specific context vector. section 12.3.1

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains weights through query-key comparisons and weighted sums, but does not detail their normalization or calculation formula.
Examples
Uses decoder positions, encoder hidden states, and local or distant amino-acid interactions as examples.
What you can do after
Understand how attention weights assemble position-specific context vectors during cross-attention.
How it connects
Connects attention weights to cross-attention, encoder hidden states, queries, keys, values, and context vectors.

Also explained in

Compare how 7 books explain attention weight →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.