A closed-book setting requires a model to answer from knowledge stored in its weights, without consulting retrieved documents during inference. It forms an alternative to retrieval-based systems. section 5.1
Closed-book systems create a distinct production tradeoff:
- They avoid retrieval latency and document-selection errors, but cannot recover information absent from the model's stored knowledge. section 5.1
- Encoder-decoder models such as T5 and BART naturally separate reading from writing, unlike decoder-only models that process one continuous prompt stream. section 5.1
Table 5.1 Classic FiD vs. big context
|
Feature
|
Classic FiD (Encoder-Decoder)
|
Long-Context RAG (Decoder-Only)
|
|
Architecture
|
T5 / ATLAS. Encodes each document
independently, then fuses them only during generation.
|
GPT / Llama. Concatenates all
documents into one giant sequence and processes them simultaneously.
|
|
Complexity
|
Linear. Encoding 100 documents
costs roughly 10 times the compute of encoding 10.
|
Quadratic. Self-attention costs
grow squarely with length. Encoding 100 documents is significantly more
expensive than encoding 10.
|
|
Lost in the Middle
|
Low Risk. Because the encoder
processes every document in isolation, it creates a distinct representation
for every passage before fusion occurs.
|
High Risk. Large Decoder-only
models often ignore information buried in the middle of a massive prompt
context.
|
|
Implementation
|
Harder. Requires fine-tuning and
hosting your own model (e.g., HuggingFace transformers).
|
Easier. Often just an API call or
a prompt template change.
|
|
Best For
|
Scale and Efficiency. When you
have a fixed domain (e.g., medical records) and need to synthesize answers
from 50–100+ documents cheaply and accurately.
|
Flexibility and Speed. When you
need to build a system today that handles general queries over a few dozen
documents without training.
|
Compare closed-book-style encoder-decoder processing with long-context decoder-only retrieval architectures across complexity, context handling, and implementation. section 5.1
