An attention score is a raw numerical compatibility value, or logit, produced by comparing a decoder query with each encoder key. It measures how relevant each encoder position is before softmax converts the scores into normalized attention weights. section 12.3.1
The score participates in cross-attention as follows:
- The decoder query represents its current informational need. section 12.3.1
- Comparisons with all encoder keys produce raw scores that measure compatibility. section 12.3.1
- Softmax normalizes the scores into weights, which determine each value's contribution to the context vector. section 12.3.1
Protein-sequence decoding
When decoding output position 45, the model might assign high raw scores to encoder positions 43–47 and position 312, if that distant position relates to a disulfide bond. Softmax then turns these scores into weights for summarizing the relevant encoder states. section 12.3.1





