liveBook · Manning

Attention

As explained in Build a DeepSeek Model (From Scratch)

Standard attention is an attention mechanism that computes interactions between query representations and key-value representations to combine relevant information.

Attention builds context-aware token representations by scoring how relevant tokens are to one another and combining their information. In next-token prediction, it lets a transformer use the preceding sequence to form a context vector for the current token. section 2.2 section 2.2.1

A single attention head follows a matrix pipeline:

  • Project the embedding matrix X with separate learned weights: Q = XWq, K = XWk, and V = XWv. Queries specify what tokens seek, keys represent what they offer, and values carry the information contributed. section 2.2.1
  • Compute scores as QKᵀ. Scale the scores and apply a causal mask and softmax to obtain attention weights; future positions receive zero weight. section 2.2.1 section 2.3.2
  • Multiply the weights by V. Each row of the resulting context matrix is a context-aware representation of its input token. section 2.2.1

Four tokens, one head

For “The next day is,” use X with shape (4, 8) and three projection matrices of shape (8, 4). Q, K, and V each have shape (4, 4); QKᵀ, the attention weights, and the context matrix also have shape (4, 4). The last context row is used to produce logits for the next token, such as “bright.” section 2.2.1 section 2.3.2

Figure 2.14 Full attention calculation for the input sequence "The next day is."

Trace the projections, scores, causal weights, and context matrix together in the four-token calculation. section 2.3.2

Multi-head attention runs separate projections and attention calculations in parallel. Each head produces its own context matrix; the matrices are concatenated and passed through a final projection. section 2.2.1

During autoregressive inference, feeding the entire growing sequence back through attention repeats earlier calculations. The query–key and attention-weight–value matrix multiplications drive quadratic growth within the attention block. A key-value cache addresses this repeated work during generation. section 2.2 section 2.3.1 section 2.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Walks through projections, scores, causal weights, context matrices, multiple heads, and repeated inference calculations with explicit tensor shapes.
Examples
Traces “The next day is” through a four-token attention head, then appends “bright” for a second inference step.
What it assumes
Introduces concepts with minimal prior-knowledge assumptions, starting from first principles.
What you can do after
Evaluate and understand attention computations and identify repeated work during autoregressive inference.
How it connects
Connects attention to causal masking, multi-head attention, key-value caching, and latent attention.

Also explained in

Compare how 40 books explain attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.