A value is the attention representation carrying the information a token contributes to context. Queries and keys determine relevance; attention weights then control how much of each value enters a context vector. section 2.2.1 section 2.4.1
Given an input embedding matrix X, a learned projection produces the value matrix V = X × Wv. Each row of V represents one token. Attention scores come from Q × Kᵀ; after scaling, causal masking, and softmax, the resulting attention weights multiply V to produce the context matrix. section 2.2.1 section 2.3.2
One attention head
For four tokens with 8-dimensional embeddings, X has shape 4×8. With Wv of shape 8×4, V has shape 4×4. The 4×4 attention-weight matrix multiplied by V produces a 4×4 context matrix: each row combines token values according to that token’s attention weights. section 2.2.1 section 2.3.2
Figure 2.6 The attention weights are multiplied by the value matrix to produce the final context matrix. Each row is now a context-aware representation of the original token.

The diagram traces attention weights through the value matrix to context-aware token representations. section 2.2.1
During autoregressive inference, computing the newest token’s context needs its attention-weight row and the full value matrix. Earlier values remain unchanged between steps and can be cached rather than projected again; the new token contributes its own value. section 2.4.1 section 2.3.2
Attention architectures can change how values are stored or shared. Grouped-query attention shares key and value projections within groups of heads, trading some diversity for lower cache memory. Multi-head latent attention instead derives V by up-projecting a compressed latent KV representation, cKV = X × Wdkv, so V = cKV × Wuv; the latent representation is the quantity cached. section 2.7.1 section 3.3.1














