An attention head is an independently parameterized submodule within multi-head attention. Each head projects input embeddings through its own query, key, and value weights, then learns to attend to relationships in a distinct representational space. section 2.6.1
The implementation keeps each head's key and value projections separate: with four heads, the layer uses Wk1–Wk4 and corresponding value projections. The resulting K1, K2, and V1, V2 can therefore encode different patterns, while each query asks a different relationship-specific question. section 2.6.1
Attention-head independence provides:
- Specialization across syntactic, semantic, or descriptive relationships. section 2.6.3
- Parallel processing of multiple interpretations through different learned projections. section 2.6.3
- A larger key-value cache, because inference stores separate key and value matrices for every head. section 2.5.1 section 2.6.1
Resolving an ambiguous relationship
In “The artist painted the portrait of a woman with a brush,” one head might connect “painted” with “brush” as an instrument, while another connects “woman” with “brush” as an attribute. Combining their context vectors preserves both interpretations. section 2.6.3
Figure 2.23 Standard Multi-Head Attention (MHA). Each of the four attention heads has its own distinct Key and Value weight matrices, indicated by the different colors. This allows each head to specialize and learn different patterns.

Use the diagram to compare four attention heads with distinct key and value weight matrices, the architectural basis for head specialization. section 2.6.1






