liveBook · Manning

Multi-head attention

As explained in Build a DeepSeek Model (From Scratch)

Multi-head self-attention computes attention over an input sequence through multiple parallel attention heads, with each head having its own learned projections.

Multi-head self-attention lets tokens weigh other tokens in a sequence through several parallel attention heads. Each head has its own learned query, key, and value projections, so it can learn a different view of token relationships. In a standard Transformer layer, attention is followed by a feed-forward network. section 1.2.1 section 3.17

For each head, project the input embeddings X into queries Q, keys K, and values V. Queries represent what tokens look for, keys what they offer, and values the information they contribute. The product QKᵀ gives token-to-token scores; masking and conversion to attention weights determine how values are combined into context vectors. Concatenate the heads’ context matrices and apply a final projection. section 2.2 section 2.2.1

Figure 2.11 The context matrices from all heads are concatenated to form a single, richer matrix, and then passed through a final projection layer.

Follow the independent head outputs as they are concatenated and projected into one context representation. section 2.2.1

A small projection

For four tokens with 8-dimensional embeddings, X has shape 4 × 8. Multiplying X by a learned 8 × 4 query matrix produces a 4 × 4 query matrix; separate key and value matrices produce corresponding projections. Multiple heads repeat this computation with their own learned projections. section 2.2.1

Parallel interpretations

In “The artist painted the portrait of a woman with a brush,” one head could learn to associate the brush with painting, while another associates it with the woman. These are illustrative possible specializations, not guaranteed head assignments. section 2.6.3

Independent keys and values make heads expressive, but they also enlarge the key-value (KV) cache used for fast inference. With fixed head size, MHA cache memory scales linearly with the number of heads. Multi-query attention shares one key/value pair across heads; grouped-query attention shares one pair per group. These alternatives reduce cache memory while trading away some independence between heads. section 2.6.1 section 2.8

For 16-bit cached values, MHA cache size in bytes is l × b × n × h × s × 2 × 2, where l is layers, b batch size, n heads, h head size, and s sequence length. The factors of two account for keys and values and for two bytes per stored value. Multi-head latent attention instead aims to reduce cache size by compressing information. section 2.6.2 section 2.7.2 section 2.8

Table 3.2 Comparison of attention types
Attention Type
Best Validation Loss
KV Cache Memory Size (MB)
Best Validation Perplexity
MHA
2.3721
48.00
10.72
MQA
2.4080
6.00
11.11
GQA
2.3776
24.00
10.78
MLA
2.3089
12.00
10.06

Compare validation loss and KV cache memory for four trained attention variants. In this experiment, MHA used 48 MB of cache; the measured trade-off is specific to the tested models. section 3.17

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through projections, attention scores, masking, context vectors, head combination, and cache-memory scaling.
Examples
Uses a four-token projection walkthrough, an ambiguous brush sentence, and a four-architecture training comparison.
What you can do after
Evaluate attention designs by tracing head computations and comparing expressivity with KV cache memory.
How it connects
Connects multi-head attention to Transformer blocks, KV caching, multi-query attention, grouped-query attention, and multi-head latent attention.

Also explained in

Compare how 25 books explain multi-head attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.