A context vector is a numerical representation of a token enriched with information from other sequence positions. In attention, each row of the context matrix is a weighted combination of value vectors; the weights determine how much each token contributes. section 2.2.1 section 2.4.1
To construct the matrix, project input embeddings X into queries Q, keys K, and values V. Compute scores with QKᵀ, then scale, apply a causal mask, and normalize them into attention weights A. The context matrix is C = AV. The mask prevents a position from attending to future positions. section 2.2.1 section 2.3.2
One row for the newest token
For “The next day is,” X has shape 4 × 8 in a simplified example, while Q, K, V, and C each have shape 4 × 4. After generating “bright,” its context vector requires a 1 × 5 attention-weight row multiplied by the full 5 × 4 value matrix, yielding a 1 × 4 vector. section 2.2.1 section 2.4.1
Figure 2.18 The context vector for "bright" is calculated by multiplying the attention weights for "bright" with the full Value matrix.

Follow the newest token’s attention-weight row as it combines the full value matrix into one context vector. section 2.4.1
In multi-head attention, each head constructs its own context matrix; the matrices are concatenated and passed through a final projection. For next-token prediction, the last position carries a weighted summary of the preceding sequence through the rest of the Transformer to the logits. section 2.2.1 section 2.2.3 section 2.3.2
Computing only the newest context vector motivates the key-value cache: its query is compared with the full key matrix, and the resulting weights combine the full value matrix. Previously computed keys and values can be reused instead of recomputed at each generation step. section 2.4.1 section 2.3.2
A context vector output by a Transformer block is also called a hidden state. The final trunk produces one hidden state per input token; these states can serve as the starting point for multi-token prediction modules. section 5.3.1


