liveBook · Manning

Encoder-decoder architecture

As explained in Retrieval Augmented Generation, The Foundational Ideas

An encoder-decoder architecture transforms input into an encoded representation and generates output conditioned on that representation.

A retrieval system can find the right passage and still need a way to turn it into a fluent answer. An encoder-decoder model separates those jobs: the encoder reads the input as a whole and builds contextual representations; the decoder generates output one token at a time, attending to those representations and to what it has already written. section 4.1.2 section 5.4

In RAG, BART supplies this generation capability. A retrieved passage is concatenated with the question and given to BART as ordinary input; changing the passage does not change the model’s weights. Unlike an encoder-only BERT-style reader, its autoregressive decoder can write a free-form answer. Cross-attention lets generation draw on encoded input rather than merely fill a blank or select a span. section 4.1.2 section 4.1.3

Fusion-in-Decoder (FiD) uses the same separation differently when many passages are available. It encodes each question–passage pair independently, then groups their representations into one memory for the decoder. The decoder can attend across that fused memory while writing an answer. This lets retrieval prioritize finding relevant passages, leaving the reader to handle distractors. section 5.1 section 5.4

Listing 5.3 Decoding for FiD.
def decode_answer(encoder_hidden_states, attention_mask, tokenizer, model):
 
    num_passages, seq_len, hidden_dim = encoder_hidden_states.shape
 
    fused_hidden_states = encoder_hidden_states.reshape(
        1, num_passages * seq_len, hidden_dim  #A
    )
    fused_attention_mask = attention_mask.reshape(1, num_passages * seq_len)
 
    encoder_outputs = BaseModelOutput(last_hidden_state=fused_hidden_states)
 
    generated_ids = model.generate(  #B
        encoder_outputs=encoder_outputs,
        attention_mask=fused_attention_mask,
        max_length=64  #C
    )
 
    return tokenizer.decode(generated_ids[0], skip_special_tokens=True)

The decoding function reshapes separate passage states and attention masks into a single decoder memory, then passes that memory to generate. section 5.4

Answering across retrieved contexts

A FiD example supplies three contexts for a question about why a project chose multi-GPU training. The encoded states have shape [3, 124, 768]; after fusion, the generated answer identifies the reason as reducing training to a feasible six-week window. The output illustrates synthesis rather than selecting a passage span. section 5.4

Choose the architecture for the reading and writing pattern required, not simply because it is more elaborate. FiD’s independent passage encoding contrasts with placing all documents in one long decoder-only prompt; it also requires fine-tuning and hosting a model. An extractive reader remains a different choice when selecting a span is sufficient. section 5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains encoder and decoder roles, cross-attention, and FiD’s tensor reshaping through an implementation.
Examples
Uses a translator analogy and a three-passage question-answering example with encoder states and a generated answer.
What it assumes
Defines encoder and decoder roles before applying them to FiD; assumes familiarity with retrieved passages and question-answering inputs.
What you can do after
Build it from scratch using Hugging Face model calls to encode passages separately and generate from fused states.
How it connects
Links retrieval-augmented generation, BART, T5, cross-attention, Fusion-in-Decoder, and extractive question answering.

Also explained in

Compare how 14 books explain encoder-decoder architecture →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.