liveBook · Manning

Closed-book setting

As explained in Retrieval Augmented Generation, The Foundational Ideas

A closed-book knowledge-task setting requires a system to answer without consulting external information during inference.

A closed-book setting requires a model to answer from knowledge stored in its weights, without consulting retrieved documents during inference. It forms an alternative to retrieval-based systems. section 5.1

Closed-book systems create a distinct production tradeoff:

  • They avoid retrieval latency and document-selection errors, but cannot recover information absent from the model's stored knowledge. section 5.1
  • Encoder-decoder models such as T5 and BART naturally separate reading from writing, unlike decoder-only models that process one continuous prompt stream. section 5.1
Table 5.1 Classic FiD vs. big context
Feature
Classic FiD (Encoder-Decoder)
Long-Context RAG (Decoder-Only)
Architecture
T5 / ATLAS. Encodes each document independently, then fuses them only during generation.
GPT / Llama. Concatenates all documents into one giant sequence and processes them simultaneously.
Complexity
Linear. Encoding 100 documents costs roughly 10 times the compute of encoding 10.
Quadratic. Self-attention costs grow squarely with length. Encoding 100 documents is significantly more expensive than encoding 10.
Lost in the Middle
Low Risk. Because the encoder processes every document in isolation, it creates a distinct representation for every passage before fusion occurs.
High Risk. Large Decoder-only models often ignore information buried in the middle of a massive prompt context.
Implementation
Harder. Requires fine-tuning and hosting your own model (e.g., HuggingFace transformers).
Easier. Often just an API call or a prompt template change.
Best For
Scale and Efficiency. When you have a fixed domain (e.g., medical records) and need to synthesize answers from 50–100+ documents cheaply and accurately.
Flexibility and Speed. When you need to build a system today that handles general queries over a few dozen documents without training.

Compare closed-book-style encoder-decoder processing with long-context decoder-only retrieval architectures across complexity, context handling, and implementation. section 5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Contrasts stored-knowledge generation with retrieval and distinguishes encoder-decoder processing from decoder-only prompt processing.
Examples
Uses T5 and BART as closed-book generative examples, contrasting them with GPT, LLaMA, and Mistral.
What you can do after
Evaluate whether stored model knowledge or retrieval-based context better fits a knowledge-answering architecture.
How it connects
Links closed-book generation to retrieval-augmented generation, encoder-decoder models, decoder-only models, T5, BART, and FiD.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.