liveBook · Manning

Cross-attention

As explained in Retrieval Augmented Generation, The Foundational Ideas

Cross-attention is an attention mechanism in which one sequence selects information from the representations of another sequence.

A retrieved answer may depend on facts scattered across several passages. Cross-attention lets an encoder–decoder generator consult encoded source text while producing each answer token. In retrieval-augmented generation, this is how the decoder selects useful information from retrieved documents rather than relying only on knowledge in its weights. section 4.1.3 section 5.4

Fusion-in-Decoder (FiD) makes cross-attention the point where multiple passages meet:

  • Pair the question with each passage and encode each pair independently. Encoder self-attention stays within that pair, avoiding self-attention over one enormous input. section 5.2
  • Reshape the separate encoder outputs into one long sequence. This fusion does not sum or merge representations; it makes all passage representations available to the decoder. section 5.2
  • At each generation step, decoder cross-attention reads that combined sequence. The decoder can select evidence across passages and disregard distractors, provided the needed evidence was retrieved. section 5.2 section 5.0

Combining evidence

Suppose a question requires a historical act, the president associated with it, and that president’s birthplace. Those clues can arrive in different retrieved passages. Joint cross-attention gives the decoder access to them while it constructs one answer; the behavior of combining evidence is learned by training the FiD reader on question–answer pairs, not obtained merely by arranging a prompt. section 5.3

Listing 5.3 Decoding for FiD.
def decode_answer(encoder_hidden_states, attention_mask, tokenizer, model):
 
    num_passages, seq_len, hidden_dim = encoder_hidden_states.shape
 
    fused_hidden_states = encoder_hidden_states.reshape(
        1, num_passages * seq_len, hidden_dim  #A
    )
    fused_attention_mask = attention_mask.reshape(1, num_passages * seq_len)
 
    encoder_outputs = BaseModelOutput(last_hidden_state=fused_hidden_states)
 
    generated_ids = model.generate(  #B
        encoder_outputs=encoder_outputs,
        attention_mask=fused_attention_mask,
        max_length=64  #C
    )
 
    return tokenizer.decode(generated_ids[0], skip_special_tokens=True)

The decoding example reshapes passage states and their attention mask into matching long sequences, then passes both to generate. The mask matters because the decoder must read the fused representation with its padding positions identified. section 5.4

This design avoids quadratic self-attention across all retrieved passages during encoding, but it does not make generation free. Every output step may read the concatenated key–value state; at scale, GPU memory bandwidth can dominate latency. Choose the passage count with both answer coverage and decoding cost in mind. section 5.2 section 5.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains FiD’s independent encoding, tensor reshaping, and decoder access at each generation step, without deriving attention mathematics.
Examples
Uses multi-passage historical reasoning and a Transformers decoding example that reshapes hidden states and attention masks.
What it assumes
Introduces encoder–decoder roles before implementation; expects enough Python and tensor familiarity to follow a reshape-based example.
What you can do after
Apply it via Transformers to expose separately encoded passages to one decoder, while accounting for training and generation latency.
How it connects
Connects encoder–decoder models, self-attention, retrieval-augmented generation, Fusion-in-Decoder, and retrieval recall.

Also explained in

Compare how 8 books explain cross-attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.