liveBook · Manning

Foundation model

As explained in Retrieval Augmented Generation, The Foundational Ideas

A foundation model is a broadly trained model that can be adapted to multiple downstream tasks.

A foundational model is a general-purpose model trained on broad, unlabeled text and designed for adaptation to downstream tasks. In retrieval-augmented training, it learns language and document retrieval together, combining a retriever with a reader or encoder. section 3.2

A retrieval-augmented foundational model develops in two stages:

  • Unsupervised pretraining trains the retriever and encoder from scratch on a corpus such as Wikipedia or CC-News using masked language modeling. section 3.2
  • Supervised fine-tuning adapts the resulting model to a task such as open-domain question answering, using labeled question–answer data and updating the reader and retriever. section 3.2
  • A secondary index builder periodically recomputes document embeddings so training can retrieve from a fresh index at scale. section 3.2
Figure 3.2 The Two-Stage Training Process of REALM. This diagram illustrates the two distinct phases of training a REALM model.

Use the two-stage training diagram to distinguish general pretraining from task-specific adaptation. section 3.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains two training stages, joint retriever-reader learning, and periodic index rebuilding, but not optimization internals.
Examples
Uses Wikipedia or CC-News pretraining and Natural Questions fine-tuning for open-domain question answering.
What it assumes
Assumes familiarity with language-model pretraining, retrieval, masked language modeling, and labeled downstream tasks.
What you can do after
Apply it via supervised fine-tuning to specialize general retrieval-augmented knowledge for a downstream task.
How it connects
Connects masked language modeling, retrieval-augmented language models, supervised fine-tuning, and open-domain question answering.

Also explained in

Compare how 30 books explain foundation model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.