liveBook · Manning

Multi-head attention

As explained in Rearchitecting LLMs

Multi-head attention is an attention mechanism that computes attention through multiple parallel representation subspaces and combines their outputs.

Multi-head attention processes an embedding through parallel heads, then concatenates their outputs into a context-aware embedding of the same dimension. In DistilGPT2, 12 heads each process a 64-dimensional subspace of a 768-dimensional embedding; the c_attn layer generates their projections together. section 3.1.2

Each head uses Query, Key, and Value vectors: a token’s Query compares with the sequence’s Keys, and softmax-normalized scores determine how much of each Value contributes to its enriched embedding. For example, “spring” can use clues from other words to resolve its context. section 3.1.2

Figure 3.3 Multi-Head Attention architecture. The input embedding (768 dimensions) is divided into 12 specialized attention heads, each processing 64 dimensions in parallel. Each head focuses on different contextual relationships, before concatenating their outputs into a context-aware embedding that incorporates information from the entire sequence. Each head focuses on different contextual relationships (labels are illustrative; real specialization emerges from training and is not directly interpretable).

The diagram shows the 768-dimensional input divided among 12 parallel 64-dimensional heads, whose outputs are concatenated. Head labels are illustrative; learned specializations are not directly interpretable. section 3.1.2

Independent Key and Value pairs make MHA’s cache grow with each processed token. Grouped-Query Attention (GQA) reduces that storage by sharing pairs among groups of Query heads; Multi-Query Attention (MQA) shares one pair across all heads. In a six-layer, 12-head example, GQA processes 32 tokens within the cache budget MHA uses for 8. section 3.1.2 section 3.2.1

How Rearchitecting LLMs covers it

How deep it goes
Explains head dimensions and Q/K/V roles, then compares MHA, GQA, and MQA cache storage.
Examples
Uses DistilGPT2’s 12 heads over 768 dimensions, a “spring” token example, and cache-budget comparisons with GQA.
What you can do after
Evaluate how head partitioning contextualizes embeddings and how shared Key and Value pairs reduce cache use.
How it connects
Connects attention heads with Query, Key, and Value projections, KV caches, Grouped-Query Attention, and Multi-Query Attention.

Also explained in

Compare how 25 books explain multi-head attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.