Cross-attention

As explained in Machine Learning Algorithms in Depth

Cross-attention computes attention with queries from one sequence or representation source and keys and values from another.

Cross-attention is multihead attention in which queries come from one representation source while keys and values come from another. In an encoder-decoder transformer, the decoder supplies queries and the encoder supplies keys and values, linking generated output positions to the input sequence. section 11.3

The encoder-decoder arrangement works as follows:

• Treat the encoder representation as a database of keys and values. section 11.3
• Use the decoder’s target sequence as the query source. section 11.3
• Associate each target word with related input words while generating the output sentence. section 11.3
• Combine this operation with masked self-attention, which preserves causality during one-token-at-a-time generation. section 11.3

Neural machine translation

While translating, the decoder queries the encoded input sentence to determine which input words are relevant to the next output word. section 11.3

Figure 11.10 Transformer architecture

Use the transformer architecture figure to locate encoder-decoder attention alongside the decoder’s other attention components. section 11.3

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines cross-attention through differing query, key, and value sources and explains its role, but does not derive numerical computation.

Examples

Uses neural machine translation, with decoder target tokens querying encoded input tokens.

What it assumes

Assumes familiarity with multihead self-attention, queries, keys, values, and encoder-decoder transformers.

What you can do after

Apply cross-attention to relate each generated target word to relevant words in an encoder-decoder transformer.

How it connects

Connects multihead attention, encoder-decoder architecture, masked self-attention, softmax, and machine translation.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)An attention mechanism that computes attention between decoder representations and representations from an encoded conditioning input.
• 

  liveBookmanning.com
  Sutskever's ListAn attention mechanism in which queries come from one sequence or representation and keys and values come from another.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchAn attention operation where queries come from one sequence or representation and keys/values come from another, enabling information transfer between them.

Compare how 4 books explain cross-attention →

Related concepts

• attentioncontrasts with · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• decoder layerused by · A decoder layer is a Transformer block that transforms target-sequence representations using masked self-attention, feed-forward processing, and optionally cross-attention to encoder outputs.
• decodeused by · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• key-value cacheuses · A key-value cache is a data structure that stores previously computed transformer attention keys and values for reuse during subsequent token generation.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.