A key is an attention representation that advertises what a token represents. A query is compared with keys to score the relevance of sequence positions; values carry the information combined into a context vector. section 2.2.1 section 2.3.2
In standard attention, the input embedding matrix X is projected through a learned weight matrix: K = XWk. Queries and values use separate learned projections. Attention scores are calculated as QKᵀ, then scaled and processed with causal softmax; the resulting weights multiply V to produce context vectors. section 2.2.1 section 2.3.2
Scoring four tokens
For “The next day is,” let X have shape 4×8 and Wk have shape 8×4. K is 4×4: one four-dimensional key per token. With Q also 4×4, QKᵀ produces a 4×4 score matrix relating each query to every key. section 2.2.1 section 2.3.2
Figure 2.19 The attention weights for "bright" are derived from its query vector and the key vectors of all tokens in the sequence.

Follow how the newest token’s query is compared with the full set of keys to produce its attention weights. section 2.4.1
During autoregressive inference, recomputing old keys at every step is wasteful: their projections have already been calculated and do not change. A key-value cache retains previous keys and values. For a new token, compute its query, key, and value, then compare its query with the full key matrix, including the new key. section 2.3.2 section 2.4.1
Attention variants change how keys are produced or shared:
- In grouped-query attention, heads within a group share key and value projections, while different groups use distinct projections. Caching fewer unique matrices saves memory while retaining diversity between groups. section 2.7.1
- In multi-head latent attention, the input is first projected into a compressed latent KV matrix, cKV = XWdkv. The key matrix is then up-projected as K = cKV Wuk; the latent matrix is the object intended for caching. section 3.3.1














