A question-answering system can have the right document in its index and still retrieve the wrong passage first. Atlas addresses this ranking failure by training a retriever to find passages that help a language model produce the answer, rather than passages that merely resemble the question. section 6.7 section 6.8
Atlas separates knowledge stored in model weights from knowledge stored in a searchable document index. Its retriever selects passages; its T5-based reader uses a Fusion-in-Decoder (FiD) structure to process each question–passage pair separately before producing an answer. External documents can be updated without retraining the language model, though retrieval adds work at inference time. section 6.1 section 6.2 section 6.7
Table 6.1 Parametric vs non-parametric knowledge
|
Feature
|
Parametric Knowledge (e.g., GPT-3)
|
Non-Parametric Knowledge (e.g., Atlas)
|
|
Knowledge Freshness
|
Static; frozen at the time of training.
|
Dynamic; updatable in near real-time.
|
|
Verifiability
|
Opaque; "black box" reasoning.
|
Transparent; responses grounded in citable sources.
|
|
Cost to Update
|
Extremely high (full model retraining).
|
Low (re-index external corpus).
|
|
Domain Specialization
|
Requires expensive fine-tuning or prompting.
|
Achieved by curating the external corpus.
|
|
Hallucination Risk
|
High when knowledge is outdated or missing.
|
Low; the model can state that knowledge is not found.
|
|
Failure Mode
|
Factual hallucination when knowledge is missing/outdated.
|
Retrieval failure; can return "knowledge not
found."
|
Compare knowledge in model weights with knowledge in an external index, especially freshness, verifiability, update cost, and failure modes. section 6.1
During pre-training, Atlas reconstructs corrupted text and judges retrieved passages by whether they make that reconstruction easier. The passage being reconstructed is excluded from retrieval so the system cannot simply fetch the answer. Perplexity Distillation uses the reader’s likelihood to train retrieval without passage-relevance labels. Because training changes document embeddings, the prebuilt index must periodically be recomputed. section 6.2 section 6.8
Table 6.2 Four papers, four answers to when the retriever trains
|
Paper
|
When the retriever trains
|
What supervises it
|
|
REALM (ch. 3)
|
During pre-training, jointly with the reader
|
The reader's loss on a masked token
|
|
RAG (ch. 4)
|
During fine-tuning, query encoder only
|
Question-answer pairs, with the document encoder frozen
|
|
FiD (ch. 5)
|
Never; the retriever is a fixed component
|
Nothing. All the learning is in the reader
|
|
Atlas (ch. 6)
|
During pre-training and fine-tuning, both halves
|
The reader's likelihood of the target, on unlabeled text
|
See how Atlas’s retriever training differs from REALM, RAG, and a fixed-retriever FiD system. section 6.2
A document present but ranked second
An analyst asks why operating margins declined. A naive retriever ranks a passage listing the margin figures above a passage containing the CFO’s explanation. Expanding retrieval to two passages finds the explanation, showing that the problem is ranking, not missing knowledge. Reader-guided training aims to rank the explanatory passage first. section 6.7
In a 64-shot Natural Questions comparison, 11B-parameter Atlas scored 42.4% Exact Match against 39.6% for 540B-parameter PaLM. That result supports using retrieval rather than model size alone for knowledge-intensive tasks; it does not remove retrieval failures or index-maintenance costs. Atlas is a 2022 T5-based system, while later approaches such as RA-DIT seek similar retriever–reader cooperation through separate fine-tuning passes. section 6.3.1 section 6.2 section 6.6
