An attention head is an independent attention mechanism that enriches each token embedding with information from the rest of the sequence. section 3.1.2
In multi-head attention:
- Heads process embeddings in parallel, each focusing on different relationships learned during training. section 3.1.2
- Their outputs are concatenated to produce context-aware embeddings; specialization labels are illustrative, not directly interpretable. section 3.1.2
Context and dimensions
The token “spring” can be contextualized differently by “water” and “mountain” or by “flowers” and “season.” DistilGPT2 has 12 heads, each projecting a 768-dimensional embedding into 64 dimensions. section 3.1.2
Figure 3.3 Multi-Head Attention architecture. The input embedding (768 dimensions) is divided into 12 specialized attention heads, each processing 64 dimensions in parallel. Each head focuses on different contextual relationships, before concatenating their outputs into a context-aware embedding that incorporates information from the entire sequence. Each head focuses on different contextual relationships (labels are illustrative; real specialization emerges from training and is not directly interpretable).
Follow the parallel head paths from the input embedding through concatenation into a context-aware embedding. section 3.1.2






