liveBook · Manning

Pretraining

As explained in Retrieval Augmented Generation, The Foundational Ideas

Pretraining is the initial training of a model on broad data before adaptation to a specialized task.

Pretraining initializes a model with broad data before specialization. In retrieval-augmented systems, it can teach a language model and a neural retriever together, rather than training the retriever as a separate component. section 3.2

REALM uses masked language modeling as a self-supervised objective. Text supplies the answer key: selected words are replaced with [MASK], and the model predicts them. The retriever searches a knowledge repository while the reader combines the retrieved document with the masked sentence. Prediction success adjusts the complete network, teaching retrieval by meaning rather than wording. section 3.2

Masking a knowledge-bearing answer

For “The monarch butterfly migrates to [MASK] for the winter,” internal knowledge may be insufficient. Retrieval can supply a passage naming Mexico, making the correct prediction more likely. Masking a named entity or date creates a useful retrieval signal; masking a function word often lets grammar solve the blank without retrieval. section 3.2

Compare pretraining strategies by their supervision and optimization target:

  • REALM learns a retriever jointly with a language model from masked-language-model performance, without labeled retrieval data. section 3.2
  • DPR pretrains retrieval representations on labeled question–answer pairs, optimizing question-to-answer-passage relevance. section 4.1.1
  • Dense retriever pretraining captures semantic question meaning but can miss passages containing exact identifiers such as part numbers. section 4.1.1
Figure 3.1 REALM's End-to-End Architecture. This diagram shows REALM's two-part system, which trains a retriever and a reader jointly.

Use the architecture diagram to trace how the retriever and reader exchange information during joint pretraining. section 3.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Descends from masked-token objectives to joint retriever-reader learning and masking choices, but does not detail optimization equations.
Examples
Uses a masked monarch-migration sentence, retrieved Wikipedia evidence, and comparisons among SBERT, DPR, and REALM.
What you can do after
Evaluate masking choices and distinguish retrieval pretraining based on self-supervised language modeling or labeled question-answer pairs.
How it connects
Connects masked language modeling, language models, neural retrieval, RAG, BERT, DPR, and semantic vector search.

Also explained in

Compare how 18 books explain pretraining →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.