An attention weight is a normalized value showing how much relative attention one token gives to each token in the sequence. It is produced by applying softmax to relevance scores, so the weights in each row sum to one. section 4.1.3
The weights determine how much each token’s value vector contributes to another token’s updated representation. A higher weight means a greater contribution; the resulting context is incorporated into hidden states. section 4.1.3
“Rises” attends to “sun”
If “sun” receives a higher weight than “the” for “rises,” more information from “sun” contributes to the updated representation of “rises.” section 4.1.3
Figure 4.5 The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

The diagram connects query-key comparisons to the attention pattern and shows how weighted values form contextual representations. section 4.1.3






