Chapter 5: Fusion-in-Decoder for multi-document processing
The introduction explains why answers spread across retrieved documents are difficult for conventional RAG and presents FiD’s separation of independent encoding and fused decoding as a scalable way to synthesize them.
After this chapter you will be able to
- Scattered evidence in RAG. Explain why retrieving relevant documents may still produce an incomplete answer when the needed facts are spread across passages rather than contained in one or two top-ranked results.
- Independent encoding and fused decoding. Describe FiD’s division of labor: it encodes retrieved documents separately, then uses decoder cross-attention across their encodings to combine facts while generating an answer. Explain the stated scaling benefit of keeping document encoding independent.
- Recall-oriented retrieval tradeoff. Explain why FiD can favor getting relevant documents somewhere into the top-k set over ranking only the most precise few results, and state the decoder’s role in handling irrelevant documents.
- Extractive retrieval limits. Describe how a retrieval-extractive system such as DrQA finds answers and why it struggles when evidence is spread across passages.
- Retrieval recall precision tradeoff. Explain why retrieving too few passages and retrieving too many pose different risks, and how a capable multi-document reader changes the division of labor.
- Long-context scaling problem. Explain why putting all retrieved documents into one long prompt can be costly and unreliable, and state the scaling idea this section attributes to FiD.
- Independent passage encoding. Describe how FiD pairs the question with each retrieved passage and encodes those inputs independently.
- Concatenation as fusion. Explain what FiD means by fusion and how separate encoder outputs become the decoder's memory.
- Cross passage answer synthesis. Trace how the decoder uses the fused representation while generating an answer token by token.
- Scaling through divided labor. Justify why FiD's separation of encoding and decoding helps with many passages, and distinguish the jobs assigned to each half.
- FiD passage scaling results. Explain what happens to FiD's answer accuracy as the number of retrieved passages rises from 10 to 100, and contrast that result with extractive readers.
- Cross passage evidence synthesis. Trace how FiD can answer the section's multi-hop question using facts from different passages, and explain why a span extractor struggles.
- Two-stage FiD fine-tuning. Describe how FiD is fine-tuned first on 25 passages per question and then briefly on 100, and explain why this reduces training cost without substantially changing performance.
- Retriever precision and recall trade-off. Distinguish the retriever's priority when the reader handles only a few documents from its priority in a FiD-based system, and explain the decoder's role in that shift.
- Question passage input preparation. Describe how one FiD example stores its question, retrieved contexts, and reference answers, and how the question is paired with each passage for encoding.
- Passage fusion for generation. Trace how separately encoded passage states are reshaped and supplied to the decoder to generate one answer.
- FiD fine tuning step. Explain why a general-purpose T5 model may be fine-tuned for a particular corpus and trace the single training step shown here.
- Evidence isolation for answer selection. Explain why FiD succeeded on the multi-GPU question: independently encoded passages let the decoder identify the passage with the precise answer despite plausible distractors.
- Learned multi-document synthesis. Distinguish selecting a fact from synthesizing an answer across passages, and explain why the Apple example’s extractive answer is an argument for training the decoder rather than a fundamental failure of FiD.
- Multi-document application fit. Distinguish tasks that benefit from FiD's multi-document synthesis from simple single-fact lookups, and describe representative applications.
- Independent encoding synthesis trade-off. Explain the trade-off between FiD's independently encoded passages and a joint representation for deep cross-document synthesis.
- Decoder memory bandwidth bottleneck. Trace why FiD inference can be slow even though the encoder accounts for most floating-point operations.
- KG-FiD passage prefiltering. Trace how KG-FiD uses relationships among passages to clean retrieved context before it reaches the standard FiD encoder and decoder.
- FiDO memory bandwidth optimization. Explain FiDO's two architectural changes and why they make inference faster without sacrificing task performance.
- RFiD rationale-guided attention. Describe how RFiD identifies passages containing answer rationales and uses that signal during generation.
- MGFiD sentence-level evidence. Trace how MGFiD narrows retrieved context from passages to important sentences before decoding.
- Optimized reader retrieval limits. Distinguish improvements to FiD's reader from problems caused by missing evidence at retrieval time.
What it builds on
- Canonical RAG vs. naive RAG (chapter 4) · Provides the canonical and naive RAG approaches contrasted with FiD here.
- The retriever (chapter 4) · The role of a retriever in supplying documents to a question-answering system.
- The hallucination challenge (chapter 1) · The problem of generated answers that are not grounded in reliable evidence.
- From retrieval to synthesis (chapter 2) · The need to turn retrieved material into a synthesized answer.
- The RAG model architecture (chapter 4) · The distinction between retrieval and generation in a RAG architecture.
- Failure points within RAG systems (chapter 1) · The taxonomy of RAG failure points referenced here.
Section by section
Introduction
The introduction explains why answers spread across retrieved documents are difficult for conventional RAG and presents FiD’s separation of independent encoding and fused decoding as a scalable way to synthesize them.
- Scattered evidence in RAG: Explain why retrieving relevant documents may still produce an incomplete answer when the needed facts are spread across passages rather than contained in one or two top-ranked results.
Why it matters: It identifies the failure FiD is meant to address: finding evidence is not the same as combining it. - Independent encoding and fused decoding: Describe FiD’s division of labor: it encodes retrieved documents separately, then uses decoder cross-attention across their encodings to combine facts while generating an answer. Explain the stated scaling benefit of keeping document encoding independent.
Why it matters: The separation between processing passages and combining them is the architectural idea the introduction asks readers to take from FiD. - Recall-oriented retrieval tradeoff: Explain why FiD can favor getting relevant documents somewhere into the top-k set over ranking only the most precise few results, and state the decoder’s role in handling irrelevant documents.
Why it matters: It explains why synthesizing across documents changes what the system needs from its retriever.
Key terms: top-k documents, FP7 (Incomplete), FP5 (Not Extracted), Fusion-in-Decoder, independent encoding, fused decoding, cross-attention, encoder, decoder, precision
5.1 The challenges of multiple document contexts
This section explains why extractive readers, closed-book generators, and long-context prompting struggle with multi-document questions, motivating FiD’s approach to scalable evidence synthesis.
- Extractive retrieval limits: Describe how a retrieval-extractive system such as DrQA finds answers and why it struggles when evidence is spread across passages.
Why it matters: It explains why retrieving more documents alone does not solve a question that requires evidence from several sources. - Retrieval recall precision tradeoff: Explain why retrieving too few passages and retrieving too many pose different risks, and how a capable multi-document reader changes the division of labor.
Why it matters: This tradeoff explains why a system might retrieve more evidence despite the noise it brings. - T5 architecture and task format: Describe why the section uses T5 to discuss FiD and how T5 frames generation tasks.
Why it matters: T5’s separation of reading and writing makes the architectural choice behind FiD easier to see. - Closed-book generation tradeoffs: Distinguish the benefit of a closed-book generative answer from the limitations of relying on knowledge stored in model weights.
Why it matters: Generation solves the literal-span constraint but does not, by itself, supply current or verifiable evidence. - Long-context scaling problem: Explain why putting all retrieved documents into one long prompt can be costly and unreliable, and state the scaling idea this section attributes to FiD.
Why it matters: A large context window does not make a single enormous prompt an efficient or dependable way to use many retrieved documents.
Key terms: retrieval-extractive pipeline, DrQA, answer spans, sparse retriever, recall, precision, retriever, reader, distractors, T5
5.2 Independent encoding, fused decoding
This section explains how FiD encodes retrieved passages separately, concatenates their token representations, and lets a decoder synthesize an answer from all of them.
- Independent passage encoding: Describe how FiD pairs the question with each retrieved passage and encodes those inputs independently.
Why it matters: Keeping passage encoding separate is the first half of FiD's division of labor. - Concatenation as fusion: Explain what FiD means by fusion and how separate encoder outputs become the decoder's memory.
Why it matters: This clarifies what is fused—and prevents confusing concatenation with combining passage meanings during encoding. - Cross passage answer synthesis: Trace how the decoder uses the fused representation while generating an answer token by token.
Why it matters: The decoder is where evidence from separate passages can inform one coherent answer. - Scaling through divided labor: Justify why FiD's separation of encoding and decoding helps with many passages, and distinguish the jobs assigned to each half.
Why it matters: FiD's scaling benefit follows from where it keeps passages separate and where it brings their evidence together.
Key terms: independent encoding, question:, title:, context:, self-attention, final hidden-state representations, fusion, concatenation, reshape, sequence length dimension
5.3 Scaling to hundreds of documents
This section shows that FiD's answer accuracy improves as it receives more passages, explains how its decoder combines evidence across them, and clarifies that this behavior requires training.
- FiD passage scaling results: Explain what happens to FiD's answer accuracy as the number of retrieved passages rises from 10 to 100, and contrast that result with extractive readers.
Why it matters: The result shows why giving FiD more retrieved evidence can help rather than simply add noise. - Exact Match versus F1: Distinguish Exact Match from F1 and explain what the reported gains indicate about FiD's answers.
Why it matters: The metric distinction makes the scaling result more precise. - Cross passage evidence synthesis: Trace how FiD can answer the section's multi-hop question using facts from different passages, and explain why a span extractor struggles.
Why it matters: Combining evidence across passages is FiD's advantage on questions whose answer requires several linked facts. - Fine tuned reader requirement: Explain why FiD's ability to sift and combine many passages is not obtained simply by arranging passages in a prompt.
Why it matters: It distinguishes adopting FiD's trained reader from merely changing how a hosted model is prompted.
Key terms: FiD, retrieved passages, Exact Match, TriviaQA, Natural Questions, extractive readers, Exact Match (EM), F1, canonical one, decoder
5.3.1 Scaling training
This section explains how FiD reduces the cost of training with many passages and how its ability to use them changes retrieval and passage-count decisions.
- Two-stage FiD fine-tuning: Describe how FiD is fine-tuned first on 25 passages per question and then briefly on 100, and explain why this reduces training cost without substantially changing performance.
Why it matters: The staged approach makes training with many passages much less expensive. - Retriever precision and recall trade-off: Distinguish the retriever's priority when the reader handles only a few documents from its priority in a FiD-based system, and explain the decoder's role in that shift.
Why it matters: The reader's capacity changes what the retrieval component should optimize. - Practical passage count selection: Explain why an application should start with a small number of retrieved passages and measure performance before increasing that number.
Why it matters: A larger retrieved set is useful only when its added context outweighs its noise.
Key terms: broad adaptation, specialization, fine-tuning, passages, retriever, reader, precision, recall, FiD decoder, re-ranker
5.4 Implementing FiD with transformer models
This section implements FiD with a T5-style encoder-decoder model, then shows how to generate answers, fine-tune on reference answers, and diagnose a model-capability failure.
- Encoder and decoder roles: Explain what the encoder and decoder each do, and why decoder cross-attention is the point where FiD can combine document representations.
Why it matters: Those two roles explain why FiD keeps passages separate during encoding but combines their information during generation. - Question passage input preparation: Describe how one FiD example stores its question, retrieved contexts, and reference answers, and how the question is paired with each passage for encoding.
Why it matters: Preparing separate question–passage inputs is what lets the encoder represent each document in light of the same question. - Passage fusion for generation: Trace how separately encoded passage states are reshaped and supplied to the decoder to generate one answer.
Why it matters: The reshape and decoder handoff are the implementation steps that turn separate document representations into one answer. - Reference answers versus inference: Distinguish what the model receives when answering a new question from how the stored answers are used in training and evaluation.
Why it matters: Keeping reference answers out of inference avoids mistaking a training or scoring field for evidence supplied to the generator. - FiD fine tuning step: Explain why a general-purpose T5 model may be fine-tuned for a particular corpus and trace the single training step shown here.
Why it matters: The training step shows how reference answers can improve the same FiD pipeline used for generation. - Reader capability and prompting: Explain what the Flan-T5 versus t5-small example shows about the limits of FiD architecture, and state the prompting option mentioned for poor results.
Why it matters: A poor answer may reflect a reader or instruction-following problem rather than a failure to retrieve or encode the right passage.
Key terms: encoder, decoder, cross-attention, question, ctxs, passages, encode_passages, encoder_hidden_states, encoder_outputs, BaseModelOutput
5.5 Case study: Finding the FiD sweet spot
The case study contrasts FiD’s success at selecting one answer-bearing passage with an untrained reader’s failure to synthesize an answer across passages.
- Evidence isolation for answer selection: Explain why FiD succeeded on the multi-GPU question: independently encoded passages let the decoder identify the passage with the precise answer despite plausible distractors.
Why it matters: It explains why success at finding one fact does not necessarily demonstrate multi-document synthesis. - Learned multi-document synthesis: Distinguish selecting a fact from synthesizing an answer across passages, and explain why the Apple example’s extractive answer is an argument for training the decoder rather than a fundamental failure of FiD.
Why it matters: It prevents a single untrained test from being mistaken for FiD’s limit as a trained architecture.
Key terms: independent encoding, distractor documents, encoded representations, multi-document synthesis, decoder, fine-tuning
5.6 Applications and performance characteristics
This section explains where FiD is useful, which RAG failures it can mitigate, and the synthesis and inference costs of its design.
- Multi-document application fit: Distinguish tasks that benefit from FiD's multi-document synthesis from simple single-fact lookups, and describe representative applications.
Why it matters: The task's need for aggregation helps determine whether FiD's added processing is worthwhile. - Incomplete and overlooked evidence: Explain how FiD addresses incomplete answers and reduces the chance of overlooking a detail in a retrieved passage.
Why it matters: These are two distinct reasons that reading across retrieved passages can improve an answer. - Independent encoding synthesis trade-off: Explain the trade-off between FiD's independently encoded passages and a joint representation for deep cross-document synthesis.
Why it matters: Scaling to more documents and synthesizing them deeply are not always the same advantage. - Decoder memory bandwidth bottleneck: Trace why FiD inference can be slow even though the encoder accounts for most floating-point operations.
Why it matters: Operation counts alone can misidentify what limits real-world FiD latency.
Key terms: multi-document synthesis, open-domain question answering, Multi-document Summarization, Fact Checking, Informed Dialogue Systems, Listwise Reranking, FP7 (Incomplete), FP5 (Not Extracted), decoder, independent encoding
5.6.1 FiD's evolution
This section explains how FiD variants filter, reweight, and focus retrieved evidence or reduce decoder memory costs, while leaving failures in initial retrieval unresolved.
- KG-FiD passage prefiltering: Trace how KG-FiD uses relationships among passages to clean retrieved context before it reaches the standard FiD encoder and decoder.
Why it matters: Cleaning context before fusion helps prevent relevant evidence from being obscured by noisy passages. - FiDO memory bandwidth optimization: Explain FiDO's two architectural changes and why they make inference faster without sacrificing task performance.
Why it matters: It shows that decoder memory access, not just model size, can determine the cost of multi-document answers. - RFiD rationale-guided attention: Describe how RFiD identifies passages containing answer rationales and uses that signal during generation.
Why it matters: The decoder can give more weight to passages that support an answer rather than merely resemble the query. - MGFiD sentence-level evidence: Trace how MGFiD narrows retrieved context from passages to important sentences before decoding.
Why it matters: Sentence-level signals focus the decoder on useful evidence while reducing its reading load. - Independent encoding and reader fusion: Explain the architectural division that persists across FiD and the extensions described here.
Why it matters: This division explains why the variants can change context selection or decoder behavior without making the retriever perform the synthesis. - Optimized reader retrieval limits: Distinguish improvements to FiD's reader from problems caused by missing evidence at retrieval time.
Why it matters: A better reader cannot fix every answer failure when the necessary evidence is unavailable to it.
Key terms: KG-FiD, Graph Neural Network (GNN), reranking, pruning, FiDO, cross-attention sparsification, parameter reallocation, memory bandwidth bottleneck, RFiD, rationale signal
5.7 Summary
Where this is used later
- (chapter intro) (chapter 13) · The challenge of giving a generator multiple document contexts.
- The parametric vs. non-parametric knowledge tradeoff (chapter 6) · Provides the closed-book side of the tradeoff between knowledge in model parameters and external knowledge.
- Types of compression (chapter 13) · Provides an example of pruning context before generation.
- The Reciprocal Rank Fusion algorithm (chapter 8) · Distinguishes combining retrieval rankings from the reader's synthesis of evidence.
- Dynamic knowledge retrieval (chapter 6) · Motivates attention to retrieval rather than reader improvements alone.
Concepts explained in this chapter
- fusion-in-decoderFused decoding generates an answer while jointly attending to representations of multiple independently encoded documents.
- independent encoding architectureAn independent encoding architecture encodes a query and each candidate document separately so their representations can be precomputed and compared efficiently.
- cross-attentionCross-attention is an attention mechanism in which one sequence selects information from the representations of another sequence.
- T5T5 is a text-to-text Transformer model family that frames diverse natural-language tasks as sequence-to-sequence text transformations.
- encoder-decoder architectureAn encoder-decoder architecture transforms input into an encoded representation and generates output conditioned on that representation.
- T5ForConditionalGenerationT5ForConditionalGeneration is a Hugging Face Transformers model class that performs conditional text generation with a T5 encoder-decoder model.
- tokenizerA tokenizer converts text into token identifiers and related tensors that a neural language model can process.
- transformersHugging Face Transformers is a library providing pretrained transformer models and tools for natural-language and other machine-learning tasks.
- decoder-only transformerA decoder-only model generates text from a single autoregressive sequence without a separate encoder for input representation.
- lexical searchLexical search retrieves text by matching words or other surface forms rather than representing meaning directly.
- long-context RAGLong-context RAG is a retrieval-augmented generation pattern that concatenates many retrieved documents into a large model input context.
- retrieval-extractive pipelineA retrieval-extractive pipeline retrieves relevant passages and then extracts a literal answer span from each passage.
- self-attentionSelf-attention is an attention mechanism that computes relationships among tokens by allowing each token to incorporate information from other tokens in the same sequence.
- synthesisSynthesis generates a novel answer by integrating information from multiple evidence sources rather than merely extracting text.
- decoderA decoder is a neural-network component that generates an output sequence from encoded representations and its previously generated tokens.
- recallRecall is the proportion of relevant items that a retrieval or classification system successfully returns.
- encoderAn encoder is a model component that transforms an input sequence into a numerical representation.
- tensorA tensor is a multidimensional numerical array used to represent data and intermediate computations in machine-learning models.
- multi-document synthesis taskMulti-document synthesis is the generation of an answer by integrating information from multiple source documents.
- FiDOFiDO is a Fusion-in-Decoder architecture optimized by reducing cross-attention and reallocating parameters toward the decoder.
- KG-FiDKG-FiD is a Fusion-in-Decoder extension that uses graph-based passage refinement before decoding.
- MGFiDMGFiD is a Fusion-in-Decoder architecture that selects fine-grained evidence and represents key sentences with an anchor vector.
- RFiDRFiD is a Fusion-in-Decoder architecture that guides decoding with scores identifying passages containing answer rationales.
- anchor vectorAn anchor vector is a numerical summary representation formed from important sentences selected from retrieved passages.
- cross-attention sparsificationCross-attention sparsification reduces the number of decoder cross-attention layers or operations to lower memory access costs.
- few-shot learningFew-shot prompting conditions a model with a small set of input-output examples that demonstrate the desired task or format.
- graph neural networkA graph neural network is a neural network that learns representations from graph-structured data by aggregating information across connected nodes.
- rationale signalA rationale signal is information passed to a decoder that indicates how strongly passages support an answer.
- latencyInference time is the time required for a trained model or system to produce an output for an input.
- exact-match accuracyExact Match is a question-answering metric that counts an answer as correct only when it exactly matches the reference answer after normalization.
- f-scoreF1 score is the harmonic mean of precision and recall.
- multi-hop reasoningMulti-hop reasoning is reasoning that connects evidence or inferences across multiple intermediate steps or sources.
- fine-tuningFine-tuning is further training of a pretrained model on task-specific or domain-specific data.
- RetrieverA retriever is a model or system component that searches an indexed collection and returns potentially relevant documents or passages.
- closed-book settingA closed-book knowledge-task setting requires a system to answer without consulting external information during inference.
- lost in the middleA language-model performance degradation caused by relevant information being placed amid distracting or competing context.
- span-corruption objectiveA span-corruption objective trains a model to reconstruct contiguous spans of tokens replaced by sentinel markers.
- pruningPruning is the removal of less relevant branches or items from a candidate set during retrieval.
- Natural QuestionsNatural Questions is a dataset of naturally occurring questions paired with answers for training and evaluating question-answering systems.
- precisionPrecision is the proportion of retrieved items that are relevant according to the evaluation criterion.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
