A query vector represents what a token seeks from other tokens in attention. It is compared with key vectors, which represent what tokens offer; the resulting attention weights determine how their value vectors contribute to a context vector. section 2.2.1 section 2.4.1
For an input embedding matrix X, a learned projection produces the query matrix Q = XWq. Separate learned projections produce K and V. The score matrix QKᵀ compares each query with the keys; scaling and causal softmax turn the scores into attention weights, which are multiplied by V. section 2.2.1 section 2.3.2
Following one new token
For “The next day is,” let X have shape (4, 8) and Wq shape (8, 4). Then Q has shape (4, 4), and QKᵀ has shape (4, 4). When “bright” arrives, its query has shape (1, 4). Multiplying it by the transposed key matrix for all five tokens gives a (1, 5) score row; its attention weights multiply the (5, 4) value matrix to produce the new token’s context vector. section 2.2.1 section 2.3.2 section 2.4.1
Figure 2.19 The attention weights for "bright" are derived from its query vector and the key vectors of all tokens in the sequence.

Trace how one new token’s query is compared with all token keys to produce its attention weights. section 2.4.1
During autoregressive inference, only the newest token’s context vector is needed for the next prediction. Its query is computed fresh, while previously computed keys and values can be reused instead of projected again. In Multi-Head Latent Attention, the new query is still projected directly, but the new token’s key-value information is first projected into a compressed latent vector. section 2.4.1 section 3.4.1













