When a model encounters a pronoun such as “it,” different tokens in the available context may contribute unequally to its representation. An attention weight is a normalized value that determines how much one token’s value contributes to that representation. section 12.2.1
Self-attention compares a query with candidate keys using dot products. These scores are divided by the square root of the key-vector dimension to prevent them from becoming excessively large. The resulting compatibility scores, or attention logits, are not yet normalized: they can be negative or positive and need not sum to one. section 12.2.1
Softmax converts each score sᵢ into a weight by dividing its exponential by the sum of exponentials across the candidate tokens. The weights sum to one. section 12.2.1
The contextual representation for “it” is a weighted sum of the candidate value vectors. A larger weight gives that value a larger contribution. section 12.2.1
Resolving a pronoun
In a simplified comparison, the query for “it” has dot products of 0.2 with the key for “trophy” and 0.8 with the key for “box.” With a key dimension of 2, the scaled scores are approximately 0.14 and 0.57. Softmax gives weights of 0.39 and 0.61. Using value vectors [1, 0] and [0, 1], respectively, the weighted sum is [0.39, 0.61]: both candidates contribute, but “box” contributes more. section 12.2.1
Direct comparison can make distant relationships available without passing information through a long chain of recurrent updates, but it does not guarantee a correct interpretation. An attention weight indicates influence within a calculation, not a complete causal explanation of the final output. Relevant information can also receive insufficient attention or be interpreted incorrectly. section 12.2.1 section 12.4.2
A single calculation is also not the whole model. Trained transformers combine many attention heads with feed-forward networks and stacked layers; simplified vectors make the mechanism visible without reproducing that full behavior. section 12.2






