An attention head is a complete attention computation that updates token representations using information from other tokens in a sequence. Rather than carrying each token’s original representation forward unchanged, it combines information according to how strongly the tokens relate to one another. section 4.1.3
Each token representation is transformed through three learned projections into a query, a key, and a value. The query represents what the token looks for in context; the key helps determine its relevance to other tokens; and the value carries the information it can contribute. Comparing queries with keys produces attention weights. Values with higher weights contribute more to the resulting representation. section 4.1.3
Figure 4.5 The model turns each token’s representation into three vectors: a query that expresses what it seeks, a key that describes characteristics used to assess relevance, and a value that carries the information it can contribute. Each is produced by its own learned projection. By comparing queries and keys, the attention head produces an attention pattern that reflects how strongly tokens are related in the model’s internal representation, with darker cells in the matrix indicating stronger connections. The values are then combined according to this pattern, giving each token a new representation, known as a hidden state, that integrates contextual information from across the sequence.

This diagram follows the computation from learned projections through attention weights to contextualized token representations. section 4.1.3
Relating a verb to its context
In “The sun rises in the east,” the query for “rises” may match the key for “sun” more strongly than the key for “the.” The value associated with “sun” then contributes more to the updated representation of “rises.” section 4.1.3
A single head provides one way of representing relationships, but a sequence can contain several kinds at once. Multi-head attention runs several heads in parallel on the same input, each with its own learned projections. Their outputs are placed side by side and mixed by a final learned transformation, producing a representation that integrates their contributions. section 4.1.4
Different heads may attend to different patterns: one might link “rises” to “sun,” while another emphasizes “east.” These roles are illustrative, not assigned in advance; the patterns emerge as the model learns. section 4.1.4






