An attention weight determines how much a token’s value vector contributes to another token’s context vector. The weights depend on the input and are computed from attention scores; they are distinct from the fixed, learned projection matrices used to create queries, keys, and values. section 2.2.1 section 2.3.2
For each attention head, compute and use the weights in four steps:
- Project input embeddings into query, key, and value matrices. Multiply Q by K transposed to obtain scores representing token-to-token relevance. section 2.3.2
- Scale the scores for training stability and apply a causal mask. Set future positions to negative infinity, not zero: softmax would give a zero score positive probability. section 2.2.1
- Apply softmax to obtain attention weights. Each row sums to 1, and masked future positions have effectively zero weight. section 2.2.1
- Multiply the attention-weight matrix by V. Each output row is a weighted sum of value vectors and forms a context-aware representation of its token. section 2.2.1
Multi-head attention repeats this calculation independently with separate query, key, and value projections. Each head produces its own weights and context matrix, allowing heads to capture different relationships. section 2.2.1
One row for the newest token
For the four-token input “The next day is,” a single head produces a 4 × 4 attention-weight matrix and a 4 × 4 context matrix. After “bright” is appended, predicting the next token requires only its context vector: multiply its 1 × 5 attention-weight row by the full 5 × 4 value matrix. That row comes from the new token’s query and the keys of all five tokens. section 2.3.2 section 2.4.1
Figure 2.18 The context vector for "bright" is calculated by multiplying the attention weights for "bright" with the full Value matrix.

Follow how one attention-weight row combines the full value matrix into the newest token’s context vector. section 2.4.1
This row-by-row view also exposes repeated work during autoregressive inference. Rather than recomputing every earlier context row, the model can calculate the newest row using the new query and the full key and value matrices, reusing cached values from previous steps. section 2.3.2 section 2.4.1






