liveBook · Manning

Context vector

As explained in Build a DeepSeek Model (From Scratch)

A context vector is a numerical representation summarizing information relevant to a model's prediction at a particular sequence position.

A context vector is a numerical representation of a token enriched with information from other sequence positions. In attention, each row of the context matrix is a weighted combination of value vectors; the weights determine how much each token contributes. section 2.2.1 section 2.4.1

To construct the matrix, project input embeddings X into queries Q, keys K, and values V. Compute scores with QKᵀ, then scale, apply a causal mask, and normalize them into attention weights A. The context matrix is C = AV. The mask prevents a position from attending to future positions. section 2.2.1 section 2.3.2

One row for the newest token

For “The next day is,” X has shape 4 × 8 in a simplified example, while Q, K, V, and C each have shape 4 × 4. After generating “bright,” its context vector requires a 1 × 5 attention-weight row multiplied by the full 5 × 4 value matrix, yielding a 1 × 4 vector. section 2.2.1 section 2.4.1

Figure 2.18 The context vector for "bright" is calculated by multiplying the attention weights for "bright" with the full Value matrix.

Follow the newest token’s attention-weight row as it combines the full value matrix into one context vector. section 2.4.1

In multi-head attention, each head constructs its own context matrix; the matrices are concatenated and passed through a final projection. For next-token prediction, the last position carries a weighted summary of the preceding sequence through the rest of the Transformer to the logits. section 2.2.1 section 2.2.3 section 2.3.2

Computing only the newest context vector motivates the key-value cache: its query is compared with the full key matrix, and the resulting weights combine the full value matrix. Previously computed keys and values can be reused instead of recomputed at each generation step. section 2.4.1 section 2.3.2

A context vector output by a Transformer block is also called a hidden state. The final trunk produces one hidden state per input token; these states can serve as the starting point for multi-token prediction modules. section 5.3.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives context matrices from attention weights and values, then traces the newest vector back to its query, keys, and values.
Examples
Works through “The next day is” and its generated token “bright” using explicit matrix dimensions.
What you can do after
Evaluate how attention constructs a context vector and why cached keys and values reduce repeated inference work.
How it connects
Connects self-attention, causal masking, multi-head attention, hidden states, and multi-token prediction.

Also explained in

Compare how 3 books explain context vector →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.