liveBook · Manning

Attention score

As explained in Build AI Drug Discovery Pipelines

An attention score is a numerical compatibility value between a query and a key before normalization into an attention weight.

An attention score is a raw numerical compatibility value, or logit, produced by comparing a decoder query with each encoder key. It measures how relevant each encoder position is before softmax converts the scores into normalized attention weights. section 12.3.1

The score participates in cross-attention as follows:

  • The decoder query represents its current informational need. section 12.3.1
  • Comparisons with all encoder keys produce raw scores that measure compatibility. section 12.3.1
  • Softmax normalizes the scores into weights, which determine each value's contribution to the context vector. section 12.3.1

Protein-sequence decoding

When decoding output position 45, the model might assign high raw scores to encoder positions 43–47 and position 312, if that distant position relates to a disulfide bond. Softmax then turns these scores into weights for summarizing the relevant encoder states. section 12.3.1

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Defines query-key compatibility and follows normalization into weights, stopping short of specifying a scoring formula.
Examples
Uses protein positions 45 and 312 to illustrate local and long-range encoder relevance.
What you can do after
Evaluate/understand how raw compatibility values become normalized contributions to a decoder context vector.
How it connects
Connects queries, keys, values, softmax, attention weights, context vectors, and cross-attention.

Also explained in

Compare how 6 books explain attention score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.