liveBook · Manning

Cross-attention

As explained in Building Specialized AI Systems

Cross-attention lets decoder positions attend to representations produced by an encoder.

Cross-attention lets decoder positions attend to representations produced by an encoder. In encoder–decoder Transformers, it connects source-sequence processing with step-by-step target generation alongside causal self-attention. section 3.1.1

Use the architecture to distinguish how attention operates:

  • The encoder builds rich, context-aware representations from the input sequence. section 3.1.1
  • The decoder generates output tokens sequentially, combining causal self-attention with cross-attention to encoder outputs. section 3.1.1
  • Encoder–decoder layouts suit sequence transduction, while encoder-only and decoder-only layouts target representation or autoregressive generation. section 3.1.1
Figure 3.2 Transformer model architecture. It consists of an encoder stack (left) and a decoder stack (right). The encoder processes the input sequence using multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks with residual connections and layer normalization (Add & LN). The decoder processes the target sequence using masked multi-head self-attention (MMSA) and multi-head cross attention (MCA) to integrate encoder features, then projects the output to probabilities through a linear layer and softmax. Source: [9]

The architecture figure locates multi-head cross-attention in the decoder between masked self-attention and output projection. section 3.1.1

How Building Specialized AI Systems covers it

How deep it goes
Explains its role within decoder blocks and contrasts it with causal self-attention, without detailing projections or training procedures.
Examples
Illustrates cross-attention with encoder–decoder language translation and a Transformer architecture diagram.
What you can do after
Evaluate how encoder–decoder Transformers combine source representations with causal target generation.
How it connects
Connects encoder outputs, decoder generation, causal self-attention, Transformer architecture, and sequence transduction.

Also explained in

Compare how 8 books explain cross-attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.