Cross-attention
Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.
Build a Text-to-Image Generator (from Scratch)
An attention mechanism that computes attention between decoder representations and representations from an encoded conditioning input.
- How it is used
- Connects decoder states to encoded source inputs during translation and image captioning.
- How deep it goes
- Specifies decoder-layer placement, query/key/value inputs, and a concrete call in code; does not elaborate the attention calculation here.
- Examples
- Uses German-to-English translation, a skiing-image caption, and a decoder-layer implementation with separate source and target masks.
- What you can do after
- Build it from scratch as a decoder sublayer that attends to encoder output for translation or image captioning.
Read how Build a Text-to-Image Generator (from Scratch) explains cross-attention →
Sutskever's List
An attention mechanism in which queries come from one sequence or representation and keys and values come from another.
- How it is used
- Aligns generated output with encoded input sequences inside an encoder-decoder generation pipeline.
- How deep it goes
- Defines Q/K/V separation and contextual alignment, but stops short of deriving attention scores or detailing implementation equations.
- Examples
- Uses translation alignment and a Transformer encoder-decoder schematic to illustrate selective source-context retrieval.
- What you can do after
- Apply it via encoder-decoder attention to guide token generation from encoded input representations.
Learn Generative AI with PyTorch
An attention operation where queries come from one sequence or representation and keys/values come from another, enabling information transfer between them.
- How it is used
- Connects decoder states to encoder representations so Transformers can generate translations conditioned on an input sequence.
- How deep it goes
- Traces queries, keys, scaled scores, softmax weights, decoder ordering, masks, residual connections, and layer normalization through code and diagrams.
- Examples
- Uses French translation tokens, encoder memory, a five-by-five target mask, attention-weight calculations, and a PyTorch DecoderLayer implementation.
- What you can do after
- Build it from scratch by assigning decoder queries and encoder keys and values within a Transformer decoder layer.
Read how Learn Generative AI with PyTorch explains cross-attention →
Machine Learning Algorithms in Depth
Cross-attention computes attention with queries from one sequence or representation source and keys and values from another.
- How it is used
- Associates decoder output positions with encoder representations during sequence generation and machine translation.
- How deep it goes
- Defines cross-attention through differing query, key, and value sources and explains its role, but does not derive numerical computation.
- Examples
- Uses neural machine translation, with decoder target tokens querying encoded input tokens.
- What you can do after
- Apply cross-attention to relate each generated target word to relevant words in an encoder-decoder transformer.
Read how Machine Learning Algorithms in Depth explains cross-attention →
Related concepts
- attentioncontrasts with · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- decoder layerused by · A decoder layer is a Transformer block that transforms target-sequence representations using masked self-attention, feed-forward processing, and optionally cross-attention to encoder outputs.
- decodeused by · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
- key-value cacheuses · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.