liveBook · Manning

Value

As explained in Build a DeepSeek Model (From Scratch)

A value is an attention representation containing the information contributed by a token when its key is selected.

A value is the attention representation carrying the information a token contributes to context. Queries and keys determine relevance; attention weights then control how much of each value enters a context vector. section 2.2.1 section 2.4.1

Given an input embedding matrix X, a learned projection produces the value matrix V = X × Wv. Each row of V represents one token. Attention scores come from Q × Kᵀ; after scaling, causal masking, and softmax, the resulting attention weights multiply V to produce the context matrix. section 2.2.1 section 2.3.2

One attention head

For four tokens with 8-dimensional embeddings, X has shape 4×8. With Wv of shape 8×4, V has shape 4×4. The 4×4 attention-weight matrix multiplied by V produces a 4×4 context matrix: each row combines token values according to that token’s attention weights. section 2.2.1 section 2.3.2

Figure 2.6 The attention weights are multiplied by the value matrix to produce the final context matrix. Each row is now a context-aware representation of the original token.

The diagram traces attention weights through the value matrix to context-aware token representations. section 2.2.1

During autoregressive inference, computing the newest token’s context needs its attention-weight row and the full value matrix. Earlier values remain unchanged between steps and can be cached rather than projected again; the new token contributes its own value. section 2.4.1 section 2.3.2

Attention architectures can change how values are stored or shared. Grouped-query attention shares key and value projections within groups of heads, trading some diversity for lower cache memory. Multi-head latent attention instead derives V by up-projecting a compressed latent KV representation, cKV = X × Wdkv, so V = cKV × Wuv; the latent representation is the quantity cached. section 2.7.1 section 3.3.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives value projection and weighted context with matrix shapes, then traces caching and alternative value representations.
Examples
Uses four-token and five-token attention calculations, plus a four-head grouped-query arrangement.
What you can do after
Evaluate and understand how values contribute to context and which computations caching avoids.
How it connects
Connects values to queries, keys, causal attention, KV caching, grouped-query attention, and multi-head latent attention.

Also explained in

Compare how 15 books explain value →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.