liveBook · Manning

Chapter 3: REALM: Birth of end-to-end trainable RAG

What you will learn in chapter 3 of Retrieval Augmented Generation, The Foundational Ideas

The introduction explains why retrieving similar documents is not enough to answer questions and presents REALM’s central shift toward training retrieval for usefulness to the reader.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains why retrieving similar documents is not enough to answer questions and presents REALM’s central shift toward training retrieval for usefulness to the reader.

Key terms: frozen, similarity-only retriever, FP2 (Missed the Top Rank), surface similarity, usefulness, REALM, joint pre-training, performance-based training, instrumental utility, encoder

3.1 Open-domain question answering

This section explains how open-domain question answering moved from keyword-based retrieval toward dense, question-aware retrieval, and why a gap between retriever and reader remained for REALM to address.

Key terms: open-domain, open-book, Open-Domain Question Answering (ODQA), retriever, reader, TF-IDF, BM25, BERT, sparse vectors, dense vector embeddings

3.2 Retrieval-augmented language model pre-training

This section explains how REALM jointly trains retrieval and reading through masked-word prediction, and how it makes that training practical.

Key terms: knowledge retriever, dual-encoder, knowledge-augmented encoder, cross-encoder, masked language modeling, self-supervised, salient span masking, marginal log-likelihood, latent variable, top-k

3.2.1 Test results

This section explains how to read retrieval and question-answering benchmarks, then uses that lens to interpret REALM’s test results and comparisons.

Key terms: accuracy, Recall@k, MRR, reader model, baseline, dataset splits, test set, ablations, accuracy-efficiency trade-off, latency

3.3 Practical implementation with Python

This section traces REALM’s retriever scores through its marginal log-likelihood loss to show how masked-word prediction trains the retriever and reader together without document labels.

Key terms: Knowledge Retriever, dual-encoder, retrieval_score, dot product, joint probability, marginal probability, marginal log-likelihood, gold_log_probs, reduce_logsumexp, performance-based training

3.4 REALM's enduring impact and legacy

This section shows how retrieved evidence improves REALM’s predictions, what REALM++ gained from stronger training, and why later RAG systems adopted a cheaper training approach.

Key terms: parametric memory, marginal likelihood, conditional probability, retriever, parametric-only models, knowledge staleness, interpretability, external document repository, modularity and updatability, REALM++

3.5 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 2: Revolutions in semantics, scale, and similarity Chapter 4: Retrieval-augmented generation for knowledge tasks →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.