Multi-head attention is a Transformer mechanism that considers several relationships among sequence elements at once. In self-attention form, multiple heads assign differing weights to positions in the same input, giving the model a broader view of how words and other elements relate. section 2.2 section 2.5
A useful mental model is many pairs of eyes reading the same story: each can focus on different characters, events, or distant passages simultaneously. This parallel focus helps capture complex, long-range dependencies that are difficult for sequential processing approaches to handle efficiently. section 2.2 section 2.5
Within a Transformer, multi-head attention contributes to a larger sequence-processing design:
- Encoder layers use multi-head self-attention while building representations of an input sequence. section 2.2
- Self-attention provides context across positions; positional encoding preserves information about their order. section 2.5
- Encoder and decoder components can be used separately or together, depending on the language task. section 2.5
Figure 2.2 The interplay of the Transformer Architecture lies in two towers, the Encoder and Decoder layers which are made up of numerous layers.
Use the architecture view to locate attention within the Encoder and Decoder towers, whose layers work together in the Transformer design. section 2.2
























