liveBook · Manning

Cross-attention

As explained in Build an Advanced RAG Application (From Scratch)

An attention mechanism in which queries from one sequence or representation attend to keys and values from another.

Cross-attention, also called encoder-decoder attention, lets a decoder focus on relevant parts of an encoded input while generating an output sequence one token at a time. section 2.3.2

A Transformer decoder layer combines three cooperating components:

  • Masked self-attention considers only previously generated tokens, preventing future-token access during generation. section 2.3.2
  • Cross-attention uses encoder representations to identify input information relevant to the next output token. section 2.3.2
  • A feed-forward neural network processes the resulting representation after the attention operations. section 2.3.2

Machine translation

For translation, the encoder produces representations of the input sequence, and the decoder generates the translated sequence step by step, using cross-attention to focus on relevant input content at each step. section 2.3.2

Cross-attention is especially useful when an encoder extracts meaningful representations and a decoder must turn them into coherent, contextually relevant text. section 2.5

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains the decoder-layer role and generation behavior, but does not detail attention equations or implementation internals.
Examples
Uses machine translation, text summarization, and text generation as application examples.
What it assumes
Assumes familiarity with encoder and decoder models while situating cross-attention alongside decoder self-attention.
What you can do after
Evaluate/understand how decoder models use encoded inputs during autoregressive text generation.
How it connects
Connects encoder representations with decoder generation, masked self-attention, feed-forward networks, and Transformer architectures.

Also explained in

Compare how 8 books explain cross-attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.