liveBook · Manning

Attention head

As explained in Machines that Think

An attention head is a complete attention computation that generates queries, keys, and values, scores token relationships, and combines values.

An attention head is a complete attention computation that updates token representations using information from other tokens in a sequence. Rather than carrying each token’s original representation forward unchanged, it combines information according to how strongly the tokens relate to one another. section 4.1.3

Each token representation is transformed through three learned projections into a query, a key, and a value. The query represents what the token looks for in context; the key helps determine its relevance to other tokens; and the value carries the information it can contribute. Comparing queries with keys produces attention weights. Values with higher weights contribute more to the resulting representation. section 4.1.3

Figure 4.5 The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

This diagram follows the computation from learned projections through attention weights to contextualized token representations. section 4.1.3

Relating a verb to its context

In “The sun rises in the east,” the query for “rises” may match the key for “sun” more strongly than the key for “the.” The value associated with “sun” then contributes more to the updated representation of “rises.” section 4.1.3

A single head provides one way of representing relationships, but a sequence can contain several kinds at once. Multi-head attention runs several heads in parallel on the same input, each with its own learned projections. Their outputs are placed side by side and mixed by a final learned transformation, producing a representation that integrates their contributions. section 4.1.4

Different heads may attend to different patterns: one might link “rises” to “sun,” while another emphasizes “east.” These roles are illustrative, not assigned in advance; the patterns emerge as the model learns. section 4.1.4

How Machines that Think covers it

How deep it goes
Traces learned projections, attention weights, and value combination through parallel heads, without formulas or implementation details.
Examples
Uses “The sun rises in the east” to illustrate differing attention to a subject and location.
What you can do after
Understand how a head combines contextual information and why multiple heads can represent different relationships.
How it connects
Connects attention heads to self-attention, hidden states, and multi-head attention.

Also explained in

Compare how 7 books explain attention head →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.