liveBook · Manning

Atlas

As explained in Retrieval Augmented Generation, The Foundational Ideas

Atlas is a retrieval-augmented language model designed for knowledge-intensive natural-language tasks.

A question-answering system can have the right document in its index and still retrieve the wrong passage first. Atlas addresses this ranking failure by training a retriever to find passages that help a language model produce the answer, rather than passages that merely resemble the question. section 6.7 section 6.8

Atlas separates knowledge stored in model weights from knowledge stored in a searchable document index. Its retriever selects passages; its T5-based reader uses a Fusion-in-Decoder (FiD) structure to process each question–passage pair separately before producing an answer. External documents can be updated without retraining the language model, though retrieval adds work at inference time. section 6.1 section 6.2 section 6.7

Table 6.1 Parametric vs non-parametric knowledge
Feature
Parametric Knowledge (e.g., GPT-3)
Non-Parametric Knowledge (e.g., Atlas)
Knowledge Freshness
Static; frozen at the time of training.
Dynamic; updatable in near real-time.
Verifiability
Opaque; "black box" reasoning.
Transparent; responses grounded in citable sources.
Cost to Update
Extremely high (full model retraining).
Low (re-index external corpus).
Domain Specialization
Requires expensive fine-tuning or prompting.
Achieved by curating the external corpus.
Hallucination Risk
High when knowledge is outdated or missing.
Low; the model can state that knowledge is not found.
Failure Mode
Factual hallucination when knowledge is missing/outdated.
Retrieval failure; can return "knowledge not found."

Compare knowledge in model weights with knowledge in an external index, especially freshness, verifiability, update cost, and failure modes. section 6.1

During pre-training, Atlas reconstructs corrupted text and judges retrieved passages by whether they make that reconstruction easier. The passage being reconstructed is excluded from retrieval so the system cannot simply fetch the answer. Perplexity Distillation uses the reader’s likelihood to train retrieval without passage-relevance labels. Because training changes document embeddings, the prebuilt index must periodically be recomputed. section 6.2 section 6.8

Table 6.2 Four papers, four answers to when the retriever trains
Paper
When the retriever trains
What supervises it
REALM (ch. 3)
During pre-training, jointly with the reader
The reader's loss on a masked token
RAG (ch. 4)
During fine-tuning, query encoder only
Question-answer pairs, with the document encoder frozen
FiD (ch. 5)
Never; the retriever is a fixed component
Nothing. All the learning is in the reader
Atlas (ch. 6)
During pre-training and fine-tuning, both halves
The reader's likelihood of the target, on unlabeled text

See how Atlas’s retriever training differs from REALM, RAG, and a fixed-retriever FiD system. section 6.2

A document present but ranked second

An analyst asks why operating margins declined. A naive retriever ranks a passage listing the margin figures above a passage containing the CFO’s explanation. Expanding retrieval to two passages finds the explanation, showing that the problem is ranking, not missing knowledge. Reader-guided training aims to rank the explanatory passage first. section 6.7

In a 64-shot Natural Questions comparison, 11B-parameter Atlas scored 42.4% Exact Match against 39.6% for 540B-parameter PaLM. That result supports using retrieval rather than model size alone for knowledge-intensive tasks; it does not remove retrieval failures or index-maintenance costs. Atlas is a 2022 T5-based system, while later approaches such as RA-DIT seek similar retriever–reader cooperation through separate fine-tuning passes. section 6.3.1 section 6.2 section 6.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains architecture, pre-training objective, leakage guard, index refresh, and reader-guided retriever training without reproducing the full joint-training pipeline.
Examples
Uses a margin-decline query with misranked passages and a 64-shot Natural Questions comparison against PaLM.
What it assumes
Introduces few-shot prompting and defines parametric versus non-parametric knowledge before explaining retriever–reader cooperation.
What you can do after
Evaluate/understand it as an architecture for improving answer-oriented retrieval, while accounting for ranking failures, index updates, and inference-time retrieval.
How it connects
Connects parametric knowledge, non-parametric knowledge, Fusion-in-Decoder, Perplexity Distillation, and few-shot learning through retriever–reader cooperation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.