Multi-head attention runs attention in parallel across several learned representation subspaces, then concatenates the outputs. Each head can focus on a different aspect of context, giving every token a richer context-aware representation. section 3.1.1
The underlying attention operation follows a compact pipeline:
- Project token embeddings into queries, keys, and values. section 3.1.1
- Compute scaled dot-product relevance scores between positions. section 3.1.1
- Apply softmax to produce attention weights and form weighted sums of values. section 3.1.1
- Concatenate the parallel head outputs before subsequent Transformer processing. section 3.1.1
Resolving context
In “The cat sat on the mat because it was tired,” self-attention can connect “it” with “the cat.” Multiple heads allow the model to attend to different contextual relationships simultaneously. section 3.1.1
























