Cross-attention lets decoder positions attend to representations produced by an encoder. In encoder–decoder Transformers, it connects source-sequence processing with step-by-step target generation alongside causal self-attention. section 3.1.1
Use the architecture to distinguish how attention operates:
- The encoder builds rich, context-aware representations from the input sequence. section 3.1.1
- The decoder generates output tokens sequentially, combining causal self-attention with cross-attention to encoder outputs. section 3.1.1
- Encoder–decoder layouts suit sequence transduction, while encoder-only and decoder-only layouts target representation or autoregressive generation. section 3.1.1
Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

The architecture figure locates multi-head cross-attention in the decoder between masked self-attention and output projection. section 3.1.1







