liveBook · Manning

Multi-head attention

As explained in Understanding AI

Multi-head attention is an attention mechanism that performs several attention operations in parallel and combines their results.

How can a model notice several kinds of relationships in the same sentence? Multi-head attention runs multiple attention processes in parallel, then combines their results. Different heads may learn to focus on grammar, meaning, or distant connections. section 2.1.3

Several readers, one sentence

When processing “it,” one head might connect it with “performance,” while others track grammatical structure or relationships between distant words. The heads’ observations combine into a richer, context-aware representation. These roles emerge through training rather than being assigned by designers. section 2.1.3

Figure 2.3 When processing "it," the model assigns an attention weight to every other token in the sentence. "Performance" receives the highest weight by far (0.68 out of 1.0)which is how the model infers what "it" refers to.

The figure shows attention weights linking “it” to other tokens, with “performance” receiving the strongest weight. section 2.1.3

How Understanding AI covers it

How deep it goes
Explains parallel heads, learned specializations, and combined results, without deriving the underlying attention calculations.
Examples
Uses pronoun resolution for “it,” its link to “performance,” and the analogy of multiple readers highlighting one sentence.
What it assumes
Assumes basic tokens and attention while relating multi-head attention to embeddings and next-token prediction.
What you can do after
Understand how parallel attention heads combine different learned relationships into context-aware representations.
How it connects
Connects attention with embeddings, next-token prediction, grammatical structure, semantic relationships, and long-range dependencies.

Also explained in

Compare how 25 books explain multi-head attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.