Multi-head attention processes an embedding through parallel heads, then concatenates their outputs into a context-aware embedding of the same dimension. In DistilGPT2, 12 heads each process a 64-dimensional subspace of a 768-dimensional embedding; the c_attn layer generates their projections together. section 3.1.2
Each head uses Query, Key, and Value vectors: a token’s Query compares with the sequence’s Keys, and softmax-normalized scores determine how much of each Value contributes to its enriched embedding. For example, “spring” can use clues from other words to resolve its context. section 3.1.2
Figure 3.3 Multi-Head Attention architecture. The input embedding (768 dimensions) is divided into 12 specialized attention heads, each processing 64 dimensions in parallel. Each head focuses on different contextual relationships, before concatenating their outputs into a context-aware embedding that incorporates information from the entire sequence. Each head focuses on different contextual relationships (labels are illustrative; real specialization emerges from training and is not directly interpretable).
The diagram shows the 768-dimensional input divided among 12 parallel 64-dimensional heads, whose outputs are concatenated. Head labels are illustrative; learned specializations are not directly interpretable. section 3.1.2
Independent Key and Value pairs make MHA’s cache grow with each processed token. Grouped-Query Attention (GQA) reduces that storage by sharing pairs among groups of Query heads; Multi-Query Attention (MQA) shares one pair across all heads. In a six-layer, 12-head example, GQA processes 32 tokens within the cache budget MHA uses for 8. section 3.1.2 section 3.2.1
























