A dot product multiplies two vectors and produces a scalar. In self-attention, stronger alignment between a token’s query and another token’s key produces a higher relevance score. section 4.1.3
These raw scores are not attention weights: the model applies softmax to each row of scores to create weights that sum to one. section 4.1.3
Comparing token representations
The query for “rises” may align more strongly with the key for “sun” than with the key for “the,” giving “sun” a higher relevance score. section 4.1.3
Figure 4.4 The query for “rises” is compared with the keys of other tokens to produce relevance scores. A stronger match with “sun” gives it a higher score than “the,” indicating that it is more useful for interpreting “rises.”

See how comparing the query for “rises” with other tokens’ keys produces different relevance scores. section 4.1.3


















