A single attention operation combines source tokens into one weighted sum. That can connect distant words, but attending to many tokens at once may wash out their individual features. Multi-head attention runs several attention computations with different learned parameters, giving the model separate ways to find useful relationships. section 15.3.1
Different heads, different clues
When matching a target word to a source sentence, one head might focus on the sentence’s subject while another focuses on punctuation. Each head produces its own output; the outputs are concatenated and projected into the final representation. section 15.3.1
The practical pattern is:
- Project queries, keys, and values separately for each head. Each head scores query–key matches, applies softmax, and combines values using those weights. section 15.3.1
- Use Keras `layers.MultiHeadAttention(num_heads=4, key_dim=16)` when a reusable layer is preferable to writing the head computations yourself. section 15.3.1 section 15.3.2
- For self-attention, pass the same sequence as query, key, and value. For cross-attention, use the target sequence as query and the source sequence as key and value. section 15.3.2 section 15.3.3
Figure 15.7 Multi-headed attention allows each target word to attend to different parts of the source sequence in separate partitions of the eventual output vector.
The diagram shows how separate heads let a target word attend to different source positions in separate parts of its output vector. section 15.3.1
In a Transformer encoder, multi-head self-attention lets each token use context from its own sequence. In a decoder, self-attention first uses the target sequence, then cross-attention brings in the encoder output. Masks keep padding out of attention and prevent decoder self-attention from seeing future target tokens. Attention alone supplies no nonlinear activation, so the encoder and decoder blocks also use feed-forward layers. section 15.3.2 section 15.3.3
