When a model encounters an ambiguous pronoun, it can compare that pronoun with surrounding tokens to build a contextual representation. In self-attention, a key is the projected representation through which a token signals its relevance to another token’s query. A value holds the information the token can contribute if attention assigns it weight. section 12.2.1 section 12.2
Each token’s embedding is transformed by a learned key projection, W_K. Separate projections produce its query and value, allowing the same token to be represented differently for seeking information, signaling relevance, and contributing content. A query–key dot product measures alignment; the resulting scores are scaled and passed through softmax to obtain attention weights. Those weights determine how values are combined. section 12.2.1
Comparing a pronoun with two keys
Suppose the query for “it” is [0.2, 0.8], and the keys for “trophy” and “box” are [1, 0] and [0, 1]. Their dot products with the query are 0.2 and 0.8. After scaling and softmax, the approximate weights are 0.39 and 0.61, so the box’s value contributes more to the contextual representation. The projections are assumed to leave the vectors unchanged; the illustrative coordinates are not literal probabilities or typical interpretable dimensions of a trained model. section 12.2.1
A single attention calculation does not account for the full behavior of a trained transformer. Many attention heads, feed-forward networks, and stacked layers operate together. section 12.2














