liveBook · Manning

T5

As explained in Retrieval Augmented Generation, The Foundational Ideas

T5 is a text-to-text Transformer model family that frames diverse natural-language tasks as sequence-to-sequence text transformations.

A question-answering system can generate a fluent answer and still get the fact wrong. T5 (Text-to-Text Transfer Transformer) is an encoder-decoder model family that generates text from text. Used alone for open-domain questions, it answers from knowledge stored in its parameters; it does not retrieve documents. That makes answers vulnerable to outdated knowledge and hallucination. section 4.5 section 5.1

On Natural Questions, T5-11B was a large closed-book baseline against retrieval-based systems. Dense passage retrieval grounded answers in documents but was limited to extracting text; RAG paired retrieval with generation and achieved higher accuracy than T5-11B. The distinction is not simply model size: retrieved evidence can supply facts that a model's weights cannot reliably retain or update. section 4.5

T5 also serves a different role as the reader in Fusion-in-Decoder (FiD). For each retrieved passage, its encoder reads that passage alongside a copy of the question. The resulting passage representations are joined as memory for one decoder, which generates an answer while attending across them. This separates passage reading from answer writing, rather than placing every document in one long prompt. section 5.1 section 5.4

Table 5.1 Classic FiD vs. big context
Feature
Classic FiD (Encoder-Decoder)
Long-Context RAG (Decoder-Only)
Architecture
T5 / ATLAS. Encodes each document independently, then fuses them only during generation.
GPT / Llama. Concatenates all documents into one giant sequence and processes them simultaneously.
Complexity
Linear. Encoding 100 documents costs roughly 10 times the compute of encoding 10.
Quadratic. Self-attention costs grow squarely with length. Encoding 100 documents is significantly more expensive than encoding 10.
Lost in the Middle
Low Risk. Because the encoder processes every document in isolation, it creates a distinct representation for every passage before fusion occurs.
High Risk. Large Decoder-only models often ignore information buried in the middle of a massive prompt context.
Implementation
Harder. Requires fine-tuning and hosting your own model (e.g., HuggingFace transformers).
Easier. Often just an API call or a prompt template change.
Best For
Scale and Efficiency. When you have a fixed domain (e.g., medical records) and need to synthesize answers from 50–100+ documents cheaply and accurately.
Flexibility and Speed. When you need to build a system today that handles general queries over a few dozen documents without training.

Compare independently encoded passages in T5-style FiD with a decoder-only system that processes a single long context; the table also identifies compute and lost-in-the-middle tradeoffs. section 5.1

A retrieved-context answer

Given a question about why a project chose multi-GPU training, a Flan-T5 FiD example encodes three passages separately and generates the answer that the strategy reduced training time to a feasible six-week window. One passage supplies the reason; another warns against confusing it with improved accuracy. section 5.4

An implementation can load a pretrained T5-family sequence-to-sequence model through Hugging Face Transformers. Pair the question with each passage, encode the resulting batch, then reshape the hidden states and attention mask from passage-level sequences into decoder memory for each question. Pass that memory to generation; fine-tuning additionally requires target answers and a training loss. T5 is a useful FiD choice when separate encoding and manageable model cost matter, but retrieval quality and answer grounding still need evaluation. section 5.4 section 5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains separate passage encoding and fused decoding, then outlines tensor reshaping and generation rather than T5's pretraining internals.
Examples
Uses Natural Questions comparisons and a three-passage Flan-T5 example answering why multi-GPU training was chosen.
What it assumes
Assumes familiarity with questions, retrieved passages, and pretrained models; develops the encoder-to-decoder FiD data flow step by step.
What you can do after
Build it from scratch using a pretrained T5-family model in Hugging Face Transformers to encode passages separately and generate a fused answer.
How it connects
Connects T5 to open-domain question answering, dense passage retrieval, RAG, Fusion-in-Decoder, and long-context decoder-only models.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.