A retrieved answer may be incomplete even when the right evidence is present: one passage gives only part of the answer, while another supplies the missing fact. Fusion-in-Decoder (FiD) lets a generative reader consider many retrieved passages together without making its encoder process them as one enormous input. section 5.0 section 5.2
For each passage, FiD pairs the question with that passage’s title and text. A T5-style encoder processes each pair independently, so its self-attention stays within one passage. The resulting representations are laid end to end—not summed—into a single sequence. At each generation step, the decoder cross-attends to that sequence to produce an answer grounded in evidence from multiple passages. section 5.2
Figure 5.2 The FiD architecture: Independent Encoding, Fused Decoding.

The architecture diagram traces question–passage pairs through separate encoding paths into the decoder’s shared view of their representations. section 5.2
This differs from an extractive reader, which selects an answer span from a passage, and from putting every document into one long encoder input. Independent encoding avoids self-attention across the full collection; it does not remove the decoder’s cost of attending to all encoded passages. The approach also changes retrieval priorities: the needed evidence can appear anywhere in the retrieved set rather than only in the first result. section 5.1 section 5.2 section 5.7
Finding one fact amid distractors
A question asks why a project chose multi-GPU training. One retrieved passage says a single GPU would make training take nine months; other passages suggest plausible but irrelevant reasons. A small Flan-T5 demonstration identifies the intended reason—reducing training to a feasible six-week window. By contrast, a test asking for several drivers of revenue growth yielded only the first passage’s fact, not a synthesis of all three. Access to multiple representations does not guarantee that the decoder will use them all. section 5.4 section 5.5
For a T5-style implementation:
- Repeat the question once per passage, tokenize the pairs, and encode them as a batch. section 5.4
- Reshape hidden states from separate passage sequences into one decoder-memory sequence; reshape the attention mask to match, then generate. section 5.4
Listing 5.3 Decoding for FiD.
def decode_answer(encoder_hidden_states, attention_mask, tokenizer, model):
num_passages, seq_len, hidden_dim = encoder_hidden_states.shape
fused_hidden_states = encoder_hidden_states.reshape(
1, num_passages * seq_len, hidden_dim #A
)
fused_attention_mask = attention_mask.reshape(1, num_passages * seq_len)
encoder_outputs = BaseModelOutput(last_hidden_state=fused_hidden_states)
generated_ids = model.generate( #B
encoder_outputs=encoder_outputs,
attention_mask=fused_attention_mask,
max_length=64 #C
)
return tokenizer.decode(generated_ids[0], skip_special_tokens=True)The decoding listing shows the critical reshape and passes both the fused hidden states and matching mask to generation. section 5.4
The original results improved as retrieval grew from 10 to 100 passages: Exact Match rose by six points on TriviaQA and 3.5 on Natural Questions. Training on that many passages was costly; a 25-passage stage followed by shorter 100-passage fine-tuning used 147 rather than 425 GPU hours with nearly identical performance. Use the separation of encoding and synthesis as a design lesson, not an automatic reason to deploy the original 2020 reader: decoder latency, reader capacity, and task-specific failures still need evaluation. section 5.3 section 5.3.1 section 5.0 section 5.7
