liveBook · Manning

Cross-attention

Also called encoder-decoder attention

Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.

How 8 Manning books explain it

Retrieval Augmented Generation, The Foundational Ideas

Cross-attention is an attention mechanism in which one sequence selects information from the representations of another sequence.

How it is used
Enables a generator to select and combine retrieved-document evidence while producing an answer.
How deep it goes
Explains FiD’s independent encoding, tensor reshaping, and decoder access at each generation step, without deriving attention mathematics.
Examples
Uses multi-passage historical reasoning and a Transformers decoding example that reshapes hidden states and attention masks.
What you can do after
Apply it via Transformers to expose separately encoded passages to one decoder, while accounting for training and generation latency.

Read how Retrieval Augmented Generation, The Foundational Ideas explains cross-attention →

Build AI Drug Discovery Pipelines

Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values derived from another.

How it is used
Introduces cross-attention as a way for a decoder to access input representations without a single-vector bottleneck.
How deep it goes
Explains query-to-key comparisons, attention weights, and the weighted context vector for each decoding step.
Examples
Uses a 500-residue protein and a possible disulfide bond between cysteines at positions 45 and 312.
What you can do after
Understand how a decoder can draw on different input positions for each output.

Read how Build AI Drug Discovery Pipelines explains cross-attention →

Build a Text-to-Image Generator (from Scratch)

An attention mechanism that computes attention between decoder representations and representations from an encoded conditioning input.

How it is used
Connects decoder states to encoded source inputs during translation and image captioning.
How deep it goes
Specifies decoder-layer placement, query/key/value inputs, and a concrete call in code; does not elaborate the attention calculation here.
Examples
Uses German-to-English translation, a skiing-image caption, and a decoder-layer implementation with separate source and target masks.
What you can do after
Build it from scratch as a decoder sublayer that attends to encoder output for translation or image captioning.

Read how Build a Text-to-Image Generator (from Scratch) explains cross-attention →

Sutskever's List

An attention mechanism in which queries come from one sequence or representation and keys and values come from another.

How it is used
Aligns generated output with encoded input sequences inside an encoder-decoder generation pipeline.
How deep it goes
Defines Q/K/V separation and contextual alignment, but stops short of deriving attention scores or detailing implementation equations.
Examples
Uses translation alignment and a Transformer encoder-decoder schematic to illustrate selective source-context retrieval.
What you can do after
Apply it via encoder-decoder attention to guide token generation from encoded input representations.

Read how Sutskever's List explains cross-attention →

Build an Advanced RAG Application (From Scratch)

An attention mechanism in which queries from one sequence or representation attend to keys and values from another.

How it is used
Guides decoder generation by connecting output-token prediction with relevant encoded input information.
How deep it goes
Explains the decoder-layer role and generation behavior, but does not detail attention equations or implementation internals.
Examples
Uses machine translation, text summarization, and text generation as application examples.
What you can do after
Evaluate/understand how decoder models use encoded inputs during autoregressive text generation.

Read how Build an Advanced RAG Application (From Scratch) explains cross-attention →

Learn Generative AI with PyTorch

An attention operation where queries come from one sequence or representation and keys/values come from another, enabling information transfer between them.

How it is used
Connects decoder states to encoder representations so Transformers can generate translations conditioned on an input sequence.
How deep it goes
Traces queries, keys, scaled scores, softmax weights, decoder ordering, masks, residual connections, and layer normalization through code and diagrams.
Examples
Uses French translation tokens, encoder memory, a five-by-five target mask, attention-weight calculations, and a PyTorch DecoderLayer implementation.
What you can do after
Build it from scratch by assigning decoder queries and encoder keys and values within a Transformer decoder layer.

Read how Learn Generative AI with PyTorch explains cross-attention →

Building Specialized AI Systems

Cross-attention lets decoder positions attend to representations produced by an encoder.

How it is used
Connects decoder generation to encoder representations in encoder–decoder sequence transduction.
How deep it goes
Explains its role within decoder blocks and contrasts it with causal self-attention, without detailing projections or training procedures.
Examples
Illustrates cross-attention with encoder–decoder language translation and a Transformer architecture diagram.
What you can do after
Evaluate how encoder–decoder Transformers combine source representations with causal target generation.

Read how Building Specialized AI Systems explains cross-attention →

Machine Learning Algorithms in Depth

Cross-attention computes attention with queries from one sequence or representation source and keys and values from another.

How it is used
Associates decoder output positions with encoder representations during sequence generation and machine translation.
How deep it goes
Defines cross-attention through differing query, key, and value sources and explains its role, but does not derive numerical computation.
Examples
Uses neural machine translation, with decoder target tokens querying encoded input tokens.
What you can do after
Apply cross-attention to relate each generated target word to relevant words in an encoder-decoder transformer.

Read how Machine Learning Algorithms in Depth explains cross-attention →

Also covered in

What to know first

Related concepts

Questions about cross-attention

What is cross-attention?

Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.

Which Manning books explain cross-attention?

Retrieval Augmented Generation, The Foundational Ideas, Build AI Drug Discovery Pipelines, Build a Text-to-Image Generator (from Scratch), Sutskever's List, Build an Advanced RAG Application (From Scratch), Learn Generative AI with PyTorch, Building Specialized AI Systems and Machine Learning Algorithms in Depth.

What should I know before learning cross-attention?

Manning books that teach cross-attention usually introduce encoder, attention, encoder-decoder architecture and softmax first.

What is the difference between cross-attention and attention?

Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.

What is cross-attention also called?

Cross-attention is also called encoder-decoder attention.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.