liveBook · Manning

Attention weight

As explained in Build a DeepSeek Model (From Scratch)

An attention weight is a context-dependent coefficient that determines how strongly an input representation contributes to an attention output.

An attention weight determines how much a token’s value vector contributes to another token’s context vector. The weights depend on the input and are computed from attention scores; they are distinct from the fixed, learned projection matrices used to create queries, keys, and values. section 2.2.1 section 2.3.2

For each attention head, compute and use the weights in four steps:

  • Project input embeddings into query, key, and value matrices. Multiply Q by K transposed to obtain scores representing token-to-token relevance. section 2.3.2
  • Scale the scores for training stability and apply a causal mask. Set future positions to negative infinity, not zero: softmax would give a zero score positive probability. section 2.2.1
  • Apply softmax to obtain attention weights. Each row sums to 1, and masked future positions have effectively zero weight. section 2.2.1
  • Multiply the attention-weight matrix by V. Each output row is a weighted sum of value vectors and forms a context-aware representation of its token. section 2.2.1

Multi-head attention repeats this calculation independently with separate query, key, and value projections. Each head produces its own weights and context matrix, allowing heads to capture different relationships. section 2.2.1

One row for the newest token

For the four-token input “The next day is,” a single head produces a 4 × 4 attention-weight matrix and a 4 × 4 context matrix. After “bright” is appended, predicting the next token requires only its context vector: multiply its 1 × 5 attention-weight row by the full 5 × 4 value matrix. That row comes from the new token’s query and the keys of all five tokens. section 2.3.2 section 2.4.1

Figure 2.18 The context vector for "bright" is calculated by multiplying the attention weights for "bright" with the full Value matrix.

Follow how one attention-weight row combines the full value matrix into the newest token’s context vector. section 2.4.1

This row-by-row view also exposes repeated work during autoregressive inference. Rather than recomputing every earlier context row, the model can calculate the newest row using the new query and the full key and value matrices, reusing cached values from previous steps. section 2.3.2 section 2.4.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces scores through scaling, causal masking, softmax, and value multiplication, including matrix shapes and independent attention heads.
Examples
Traces a four-token attention matrix and a five-token newest-row calculation using “The next day is bright.”
What you can do after
Evaluate and understand how attention weights produce context vectors and why only the newest row is needed during generation.
How it connects
Connects attention weights to causal masking, softmax, multi-head attention, context vectors, and key-value caching.

Also explained in

Compare how 7 books explain attention weight →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.