A retrieval system can find the right passage and still need a way to turn it into a fluent answer. An encoder-decoder model separates those jobs: the encoder reads the input as a whole and builds contextual representations; the decoder generates output one token at a time, attending to those representations and to what it has already written. section 4.1.2 section 5.4
In RAG, BART supplies this generation capability. A retrieved passage is concatenated with the question and given to BART as ordinary input; changing the passage does not change the model’s weights. Unlike an encoder-only BERT-style reader, its autoregressive decoder can write a free-form answer. Cross-attention lets generation draw on encoded input rather than merely fill a blank or select a span. section 4.1.2 section 4.1.3
Fusion-in-Decoder (FiD) uses the same separation differently when many passages are available. It encodes each question–passage pair independently, then groups their representations into one memory for the decoder. The decoder can attend across that fused memory while writing an answer. This lets retrieval prioritize finding relevant passages, leaving the reader to handle distractors. section 5.1 section 5.4
Listing 5.3 Decoding for FiD.
def decode_answer(encoder_hidden_states, attention_mask, tokenizer, model):
num_passages, seq_len, hidden_dim = encoder_hidden_states.shape
fused_hidden_states = encoder_hidden_states.reshape(
1, num_passages * seq_len, hidden_dim #A
)
fused_attention_mask = attention_mask.reshape(1, num_passages * seq_len)
encoder_outputs = BaseModelOutput(last_hidden_state=fused_hidden_states)
generated_ids = model.generate( #B
encoder_outputs=encoder_outputs,
attention_mask=fused_attention_mask,
max_length=64 #C
)
return tokenizer.decode(generated_ids[0], skip_special_tokens=True)The decoding function reshapes separate passage states and attention masks into a single decoder memory, then passes that memory to generate. section 5.4
Answering across retrieved contexts
A FiD example supplies three contexts for a question about why a project chose multi-GPU training. The encoded states have shape [3, 124, 768]; after fusion, the generated answer identifies the reason as reducing training to a feasible six-week window. The output illustrates synthesis rather than selecting a passage span. section 5.4
Choose the architecture for the reading and writing pattern required, not simply because it is more elaborate. FiD’s independent passage encoding contrasts with placing all documents in one long decoder-only prompt; it also requires fine-tuning and hosting a model. An extractive reader remains a different choice when selecting a span is sufficient. section 5.1













