Cross-attention, also called encoder-decoder attention, lets a decoder focus on relevant parts of an encoded input while generating an output sequence one token at a time. section 2.3.2
A Transformer decoder layer combines three cooperating components:
- Masked self-attention considers only previously generated tokens, preventing future-token access during generation. section 2.3.2
- Cross-attention uses encoder representations to identify input information relevant to the next output token. section 2.3.2
- A feed-forward neural network processes the resulting representation after the attention operations. section 2.3.2
Machine translation
For translation, the encoder produces representations of the input sequence, and the decoder generates the translated sequence step by step, using cross-attention to focus on relevant input content at each step. section 2.3.2
Cross-attention is especially useful when an encoder extracts meaningful representations and a decoder must turn them into coherent, contextually relevant text. section 2.5







