A single attention calculation produces one pattern of weights over relevant information. Multi-head attention computes several patterns in parallel: each head uses its own learned query, key, and value projections, and their outputs are combined into a richer representation. section 12.3.3
The heads let a token be considered through several relationships at once. Training determines how their work is distributed; heads are not assigned fixed roles, and some may learn overlapping or complementary patterns. Their contributions can include grammatical, semantic, positional, or referential relationships. section 12.3.3
Several relationships at once
In translation, a word can be influenced by its grammatical role, nearby phrase, and a distant term needed to preserve meaning. In code, a token can relate to syntax, variable use, function structure, and dependencies elsewhere in the program. section 12.3.3
Multi-head attention expands the relationships self-attention can represent; it does not alone explain a model’s capabilities. Its benefits work alongside other learned components, training data, and computation. section 12.3.3 section 12.6
























