Searching a large document collection is too slow if every query must be processed jointly with every document. Independent encoding separates that work: a bi-encoder turns the query and each document into representations that can be compared. Document embeddings can be computed before queries arrive, then searched with a vector index. This makes interactive retrieval practical, but fast retrieval does not by itself guarantee that the best evidence was found. section 2.3.4
Independent encoding also has a different role after retrieval. A question may need facts from several passages, while a generator reading one passage at a time produces an incomplete answer. Fusion-in-Decoder (FiD) pairs the same question separately with each retrieved passage, encodes those inputs in parallel, then lets one decoder attend to all their representations while generating an answer. Unlike indexed document embeddings, these passage representations depend on the current question; the method preserves separate reading before combining evidence. section 5.0 section 5.2
The two uses place different work offline and at query time:
- For retrieval, precompute document embeddings; encode an incoming query and use similarity search to find candidates. Sentence-BERT-style independent encoding supports this latency-oriented design. section 2.3.4
- For FiD, retrieve passages first. Build one input per passage containing the question, title, and passage text; run the encoder separately on each input. These encodings must be produced for that question. section 5.2
- Reshape the separate encoder outputs into one sequence for decoder cross-attention. Fusion does not sum the representations: the decoder reads the concatenated states to select or combine evidence. section 5.2
Listing 5.3 Decoding for FiD.
def decode_answer(encoder_hidden_states, attention_mask, tokenizer, model):
num_passages, seq_len, hidden_dim = encoder_hidden_states.shape
fused_hidden_states = encoder_hidden_states.reshape(
1, num_passages * seq_len, hidden_dim #A
)
fused_attention_mask = attention_mask.reshape(1, num_passages * seq_len)
encoder_outputs = BaseModelOutput(last_hidden_state=fused_hidden_states)
generated_ids = model.generate( #B
encoder_outputs=encoder_outputs,
attention_mask=fused_attention_mask,
max_length=64 #C
)
return tokenizer.decode(generated_ids[0], skip_special_tokens=True)The decoding function reshapes passage states and their attention mask before passing the fused states to generation. section 5.4
A fact among distractors
Asked why a project chose multi-GPU training, an example FiD implementation identifies the passage giving the six-week training-time reason despite plausible distractors. That is easier than a question requiring genuine synthesis across passages: an unfine-tuned small model instead copied a dominant passage in a revenue-growth example. Fine-tuning is needed to teach the decoder to combine evidence rather than merely favor the strongest signal. section 5.4 section 5.5
Choose the architecture for the bottleneck at hand. Precomputed independent embeddings address retrieval speed; FiD-style independent encoding addresses reading many retrieved passages without concatenating their raw text into one large input. Neither removes the need to retrieve relevant evidence. FiD also pays for fusion during generation: decoder cross-attention repeatedly reads the concatenated states, creating a memory-bandwidth and latency limit. For deep joint reasoning, a shared representation may be preferable when its cost is acceptable. section 2.3.4 section 5.6
