A self-attention matrix arranges token-to-token attention weights across a sequence. Each row gives one token’s relative attention to the others: softmax converts its relevance scores into weights that sum to one. These weights are then applied to value vectors, determining how much information each token contributes to updated, contextual representations. section 4.1.3
“The sun rises”
For “rises,” a higher weight on “sun” than on “the” means more information from “sun” contributes to its updated representation. section 4.1.3
Figure 4.5 The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

The figure shows how query-key comparisons produce the attention pattern and how weighted values form contextualized hidden states. section 4.1.3




