Multi-head self-attention lets tokens weigh other tokens in a sequence through several parallel attention heads. Each head has its own learned query, key, and value projections, so it can learn a different view of token relationships. In a standard Transformer layer, attention is followed by a feed-forward network. section 1.2.1 section 3.17
For each head, project the input embeddings X into queries Q, keys K, and values V. Queries represent what tokens look for, keys what they offer, and values the information they contribute. The product QKᵀ gives token-to-token scores; masking and conversion to attention weights determine how values are combined into context vectors. Concatenate the heads’ context matrices and apply a final projection. section 2.2 section 2.2.1
Figure 2.11 The context matrices from all heads are concatenated to form a single, richer matrix, and then passed through a final projection layer.

Follow the independent head outputs as they are concatenated and projected into one context representation. section 2.2.1
A small projection
For four tokens with 8-dimensional embeddings, X has shape 4 × 8. Multiplying X by a learned 8 × 4 query matrix produces a 4 × 4 query matrix; separate key and value matrices produce corresponding projections. Multiple heads repeat this computation with their own learned projections. section 2.2.1
Parallel interpretations
In “The artist painted the portrait of a woman with a brush,” one head could learn to associate the brush with painting, while another associates it with the woman. These are illustrative possible specializations, not guaranteed head assignments. section 2.6.3
Independent keys and values make heads expressive, but they also enlarge the key-value (KV) cache used for fast inference. With fixed head size, MHA cache memory scales linearly with the number of heads. Multi-query attention shares one key/value pair across heads; grouped-query attention shares one pair per group. These alternatives reduce cache memory while trading away some independence between heads. section 2.6.1 section 2.8
For 16-bit cached values, MHA cache size in bytes is l × b × n × h × s × 2 × 2, where l is layers, b batch size, n heads, h head size, and s sequence length. The factors of two account for keys and values and for two bytes per stored value. Multi-head latent attention instead aims to reduce cache size by compressing information. section 2.6.2 section 2.7.2 section 2.8
Table 3.2 Comparison of attention types
|
Attention Type
|
Best Validation Loss
|
KV Cache Memory Size (MB)
|
Best Validation Perplexity
|
|
MHA
|
2.3721
|
48.00
|
10.72
|
|
MQA
|
2.4080
|
6.00
|
11.11
|
|
GQA
|
2.3776
|
24.00
|
10.78
|
|
MLA
|
2.3089
|
12.00
|
10.06
|
Compare validation loss and KV cache memory for four trained attention variants. In this experiment, MHA used 48 MB of cache; the measured trade-off is specific to the tested models. section 3.17
























