liveBook · Manning

Chapter 6: Atlas: Few-shot learning with retrieval augmentation

What you will learn in chapter 6 of Retrieval Augmented Generation, The Foundational Ideas

Introduces Atlas as a way to achieve strong few-shot performance by aligning a retriever and reader around external knowledge rather than relying on ever-larger models.

After this chapter you will be able to

What it builds on

Section by section

Introduction

Introduces Atlas as a way to achieve strong few-shot performance by aligning a retriever and reader around external knowledge rather than relying on ever-larger models.

Key terms: few-shot learning, N-shot learning, zero-shot, scaling laws, retriever, reader, jointly pre-trained, supervisory signal, Advanced RAG, parametric

6.1 The parametric vs. non-parametric knowledge tradeoff

This section contrasts knowledge stored in model weights with knowledge retrieved from an editable corpus, then describes how Atlas’s retriever and reader use the latter approach.

Key terms: parametric knowledge, non-parametric knowledge, model weights, external index, external knowledge corpus, verifiable, hallucination, inference time, Contriever, dual-encoder

6.2 Atlas pre-training methodology

This section explains Atlas’s pre-training task, the safeguards and index refreshes it requires, and how its FiD reader handles many retrieved passages.

Key terms: masked language modeling, prefix-MLM, retriever, index, index-staleness problem, document embeddings, Fusion-in-Decoder (FiD), self-attention, document-query pair, cross-attention

6.2.1 Joint training without labels

This section explains how Atlas uses the reader’s answer performance to improve retrieval without labeled query-document pairs, unlike the static retriever in original FiD.

Key terms: FiD, retriever, reader, Atlas, latent variables, perplexity, relevance labels

6.2.2 The training objectives

This section explains how Atlas trains its retriever with Perplexity Distillation, contrasts three alternative losses, and interprets an ablation of their few-shot performance.

Key terms: Perplexity Distillation, PDist, top-k documents, log-probability, softmax, KL divergence, Attention Distillation, ADist, EMDR², LOOP

6.2.3 Code in practice

A small PDist demonstration shows how a reader’s answer probabilities train a retriever to favor useful passages over merely similar ones.

Key terms: reader, log-probability, target weight, softmax, relevance labels, PDist, cosine similarity, KL divergence, reader's distribution, retriever's distribution

6.3 Dynamic knowledge retrieval

This section frames how to judge Atlas’s joint training and states its reported performance advantage in few-shot settings.

Key terms: Atlas, retrieval-augmented model, parametric-only models, few-shot settings

6.3.1 Head-to-head

Head-to-head benchmark results show that Atlas achieves stronger few-shot performance than much larger models while using far fewer resources.

Key terms: 64-shot, Natural Questions, Exact Match (EM), parametric knowledge, retrieval system, TriviaQA, MMLU, GPT-3

6.3.2 The temporal sensitivity experiment

The section uses a time-swapped Wikipedia index to show that Atlas draws answers from retrieved knowledge at inference time, then explains the operational risk of depending on that index.

Key terms: temporal sensitivity, external knowledge index, inference time, parametric memory, non-parametric knowledge, index, parametric fallback, coverage audits, source curation, refresh pipelines

6.4 Efficient knowledge base updating

This section explains why updating a retriever’s document encoder makes a large knowledge-base index stale and expensive to rebuild.

Key terms: document encoder, vector embeddings, index staleness, re-indexing

6.4.1 Three strategies for efficient updates

Compares three ways to keep retrieval useful as Atlas is updated, and explains why updating only the query encoder can be best in a few-shot setting.

Key terms: full index update, re-ranking, stale index, candidate documents, query-side fine-tuning, document encoder, query encoder, document embeddings, 64-shot setting, ablation study

6.4.2 Compressing the index

This section explains how Atlas compresses a large dense-retrieval index to make single-GPU deployment practical.

Key terms: embedding index, Product Quantization (PQ), subvectors, learned codebook, retrieval quality

6.5 Few-shot learning strategies for RAG systems

This section turns Atlas’s few-shot findings into five choices for building and deploying a RAG system with limited labeled data.

Key terms: reader, Fusion-in-Decoder (FiD), multi-document processing, few-shot, retriever, jointly pre-trained, self-supervised, Perplexity Distillation (PDist), perplexity, supervisory signal

6.6 Joint training after Atlas

This section connects Atlas’s joint retriever–reader pre-training to later efforts to achieve similar cooperation more cheaply through fine-tuning.

Key terms: Atlas, retriever, reader, joint pre-training, fine-tuning, RA-DIT

6.6.1 RA-DIT: The same two components, tuned separately

RA-DIT adapts an off-the-shelf retriever and language model to work together through two separate fine-tuning passes rather than joint pre-training.

Key terms: RA-DIT, Atlas, fine-tuning passes, LM-Supervised Retrieval, KL divergence, query encoder, document encoder, Perplexity Distillation, language-model pass, instruction data

6.6.2 When you do not own the reader

This section explains how retriever training changes when the reader is a closed API model, and contrasts reader-likelihood training with alternatives that use different signals.

Key terms: REPLUG, reader, retriever, model's likelihood of the target, reader's likelihood of the correct output, ownership, open-weight models, index rebuilds, Search-R1, ReSearch

6.7 Case study: Enterprise knowledge adaptation

An enterprise earnings-report case study shows how Atlas-style training corrects a naive RAG ranking failure and adapts to new reports.

Key terms: naive RAG, cosine similarity, bi-encoder, ranking failure, Atlas-style model, joint pre-training, Perplexity Distillation (PDist), few-shot fine-tuning, retriever, reader

6.8 Summary

Concepts explained in this chapter

← Chapter 5: Fusion-in-Decoder for multi-document processing Chapter 7: HyDE: Imagining the answer before you search →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.