A foundational model is a general-purpose model trained on broad, unlabeled text and designed for adaptation to downstream tasks. In retrieval-augmented training, it learns language and document retrieval together, combining a retriever with a reader or encoder. section 3.2
A retrieval-augmented foundational model develops in two stages:
- Unsupervised pretraining trains the retriever and encoder from scratch on a corpus such as Wikipedia or CC-News using masked language modeling. section 3.2
- Supervised fine-tuning adapts the resulting model to a task such as open-domain question answering, using labeled question–answer data and updating the reader and retriever. section 3.2
- A secondary index builder periodically recomputes document embeddings so training can retrieve from a fresh index at scale. section 3.2
Figure 3.2 The Two-Stage Training Process of REALM. This diagram illustrates the two distinct phases of training a REALM model.

Use the two-stage training diagram to distinguish general pretraining from task-specific adaptation. section 3.2





























