liveBook · Manning

Multi-head attention

As explained in Timeless Algorithms, The Foundational Ideas

An attention mechanism that computes multiple attention patterns in parallel using distinct learned projections and combines their outputs.

A single attention calculation produces one pattern of weights over relevant information. Multi-head attention computes several patterns in parallel: each head uses its own learned query, key, and value projections, and their outputs are combined into a richer representation. section 12.3.3

The heads let a token be considered through several relationships at once. Training determines how their work is distributed; heads are not assigned fixed roles, and some may learn overlapping or complementary patterns. Their contributions can include grammatical, semantic, positional, or referential relationships. section 12.3.3

Several relationships at once

In translation, a word can be influenced by its grammatical role, nearby phrase, and a distant term needed to preserve meaning. In code, a token can relate to syntax, variable use, function structure, and dependencies elsewhere in the program. section 12.3.3

Multi-head attention expands the relationships self-attention can represent; it does not alone explain a model’s capabilities. Its benefits work alongside other learned components, training data, and computation. section 12.3.3 section 12.6

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains learned projections, combined head outputs, possible head roles, and their limits of interpretability.
Examples
Illustrates translation relationships and code dependencies that can influence a token simultaneously.
What you can do after
Understand how learned attention heads combine multiple relational patterns rather than relying on one dominant relationship.
How it connects
Connects self-attention and transformers with recurrent neural networks as contrasting approaches to sequence modeling.

Also explained in

Compare how 25 books explain multi-head attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.