Attention builds context-aware token representations by scoring how relevant tokens are to one another and combining their information. In next-token prediction, it lets a transformer use the preceding sequence to form a context vector for the current token. section 2.2 section 2.2.1
A single attention head follows a matrix pipeline:
- Project the embedding matrix X with separate learned weights: Q = XWq, K = XWk, and V = XWv. Queries specify what tokens seek, keys represent what they offer, and values carry the information contributed. section 2.2.1
- Compute scores as QKᵀ. Scale the scores and apply a causal mask and softmax to obtain attention weights; future positions receive zero weight. section 2.2.1 section 2.3.2
- Multiply the weights by V. Each row of the resulting context matrix is a context-aware representation of its input token. section 2.2.1
Four tokens, one head
For “The next day is,” use X with shape (4, 8) and three projection matrices of shape (8, 4). Q, K, and V each have shape (4, 4); QKᵀ, the attention weights, and the context matrix also have shape (4, 4). The last context row is used to produce logits for the next token, such as “bright.” section 2.2.1 section 2.3.2
Figure 2.14 Full attention calculation for the input sequence "The next day is."

Trace the projections, scores, causal weights, and context matrix together in the four-token calculation. section 2.3.2
Multi-head attention runs separate projections and attention calculations in parallel. Each head produces its own context matrix; the matrices are concatenated and passed through a final projection. section 2.2.1
During autoregressive inference, feeding the entire growing sequence back through attention repeats earlier calculations. The query–key and attention-weight–value matrix multiplications drive quadratic growth within the attention block. A key-value cache addresses this repeated work during generation. section 2.2 section 2.3.1 section 2.3.2







































