A question-answering system can generate a fluent answer and still get the fact wrong. T5 (Text-to-Text Transfer Transformer) is an encoder-decoder model family that generates text from text. Used alone for open-domain questions, it answers from knowledge stored in its parameters; it does not retrieve documents. That makes answers vulnerable to outdated knowledge and hallucination. section 4.5 section 5.1
On Natural Questions, T5-11B was a large closed-book baseline against retrieval-based systems. Dense passage retrieval grounded answers in documents but was limited to extracting text; RAG paired retrieval with generation and achieved higher accuracy than T5-11B. The distinction is not simply model size: retrieved evidence can supply facts that a model's weights cannot reliably retain or update. section 4.5
T5 also serves a different role as the reader in Fusion-in-Decoder (FiD). For each retrieved passage, its encoder reads that passage alongside a copy of the question. The resulting passage representations are joined as memory for one decoder, which generates an answer while attending across them. This separates passage reading from answer writing, rather than placing every document in one long prompt. section 5.1 section 5.4
Table 5.1 Classic FiD vs. big context
|
Feature
|
Classic FiD (Encoder-Decoder)
|
Long-Context RAG (Decoder-Only)
|
|
Architecture
|
T5 / ATLAS. Encodes each document
independently, then fuses them only during generation.
|
GPT / Llama. Concatenates all
documents into one giant sequence and processes them simultaneously.
|
|
Complexity
|
Linear. Encoding 100 documents
costs roughly 10 times the compute of encoding 10.
|
Quadratic. Self-attention costs
grow squarely with length. Encoding 100 documents is significantly more
expensive than encoding 10.
|
|
Lost in the Middle
|
Low Risk. Because the encoder
processes every document in isolation, it creates a distinct representation
for every passage before fusion occurs.
|
High Risk. Large Decoder-only
models often ignore information buried in the middle of a massive prompt
context.
|
|
Implementation
|
Harder. Requires fine-tuning and
hosting your own model (e.g., HuggingFace transformers).
|
Easier. Often just an API call or
a prompt template change.
|
|
Best For
|
Scale and Efficiency. When you
have a fixed domain (e.g., medical records) and need to synthesize answers
from 50–100+ documents cheaply and accurately.
|
Flexibility and Speed. When you
need to build a system today that handles general queries over a few dozen
documents without training.
|
Compare independently encoded passages in T5-style FiD with a decoder-only system that processes a single long context; the table also identifies compute and lost-in-the-middle tradeoffs. section 5.1
A retrieved-context answer
Given a question about why a project chose multi-GPU training, a Flan-T5 FiD example encodes three passages separately and generates the answer that the strategy reduced training time to a feasible six-week window. One passage supplies the reason; another warns against confusing it with improved accuracy. section 5.4
An implementation can load a pretrained T5-family sequence-to-sequence model through Hugging Face Transformers. Pair the question with each passage, encode the resulting batch, then reshape the hidden states and attention mask from passage-level sequences into decoder memory for each question. Pass that memory to generation; fine-tuning additionally requires target answers and a training loss. T5 is a useful FiD choice when separate encoding and manageable model cost matter, but retrieval quality and answer grounding still need evaluation. section 5.4 section 5.1
