liveBook · Manning

Key

As explained in Build a DeepSeek Model (From Scratch)

A key is an attention representation against which a query is compared to determine the relevance of a sequence position.

A key is an attention representation that advertises what a token represents. A query is compared with keys to score the relevance of sequence positions; values carry the information combined into a context vector. section 2.2.1 section 2.3.2

In standard attention, the input embedding matrix X is projected through a learned weight matrix: K = XWk. Queries and values use separate learned projections. Attention scores are calculated as QKᵀ, then scaled and processed with causal softmax; the resulting weights multiply V to produce context vectors. section 2.2.1 section 2.3.2

Scoring four tokens

For “The next day is,” let X have shape 4×8 and Wk have shape 8×4. K is 4×4: one four-dimensional key per token. With Q also 4×4, QKᵀ produces a 4×4 score matrix relating each query to every key. section 2.2.1 section 2.3.2

Figure 2.19 The attention weights for "bright" are derived from its query vector and the key vectors of all tokens in the sequence.

Follow how the newest token’s query is compared with the full set of keys to produce its attention weights. section 2.4.1

During autoregressive inference, recomputing old keys at every step is wasteful: their projections have already been calculated and do not change. A key-value cache retains previous keys and values. For a new token, compute its query, key, and value, then compare its query with the full key matrix, including the new key. section 2.3.2 section 2.4.1

Attention variants change how keys are produced or shared:

  • In grouped-query attention, heads within a group share key and value projections, while different groups use distinct projections. Caching fewer unique matrices saves memory while retaining diversity between groups. section 2.7.1
  • In multi-head latent attention, the input is first projected into a compressed latent KV matrix, cKV = XWdkv. The key matrix is then up-projected as K = cKV Wuk; the latent matrix is the object intended for caching. section 3.3.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Derives key projections, score matrix shapes, and cache requirements, then outlines grouped and latent alternatives.
Examples
Traces four-token matrix dimensions and the new-token query for “bright.”
What you can do after
Evaluate and understand how key reuse and sharing reduce repeated computation or cached matrices.
How it connects
Connects keys to queries, values, attention scores, causal masking, and the key-value cache.

Also explained in

Compare how 15 books explain key →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.