Pretraining initializes a model with broad data before specialization. In retrieval-augmented systems, it can teach a language model and a neural retriever together, rather than training the retriever as a separate component. section 3.2
REALM uses masked language modeling as a self-supervised objective. Text supplies the answer key: selected words are replaced with [MASK], and the model predicts them. The retriever searches a knowledge repository while the reader combines the retrieved document with the masked sentence. Prediction success adjusts the complete network, teaching retrieval by meaning rather than wording. section 3.2
Masking a knowledge-bearing answer
For “The monarch butterfly migrates to [MASK] for the winter,” internal knowledge may be insufficient. Retrieval can supply a passage naming Mexico, making the correct prediction more likely. Masking a named entity or date creates a useful retrieval signal; masking a function word often lets grammar solve the blank without retrieval. section 3.2
Compare pretraining strategies by their supervision and optimization target:
- REALM learns a retriever jointly with a language model from masked-language-model performance, without labeled retrieval data. section 3.2
- DPR pretrains retrieval representations on labeled question–answer pairs, optimizing question-to-answer-passage relevance. section 4.1.1
- Dense retriever pretraining captures semantic question meaning but can miss passages containing exact identifiers such as part numbers. section 4.1.1
Figure 3.1 REALM's End-to-End Architecture. This diagram shows REALM's two-part system, which trains a retriever and a reader jointly.

Use the architecture diagram to trace how the retriever and reader exchange information during joint pretraining. section 3.2

















