liveBook · Manning

Inverse cloze task

As explained in Retrieval Augmented Generation, The Foundational Ideas

The Inverse Cloze Task trains a retriever to match a sentence with the passage from which that sentence was removed.

A randomly initialized retriever fetches unhelpful documents, leaving reader supervision without a useful signal. The Inverse Cloze Task warm-starts retrieval before joint training by matching a sentence with the passage from which it was removed. section 3.2

The task trains a dual-encoder retriever to place a sentence and its source passage close in embedding space. This gives retrieval a partial capability that downstream masked-language-modeling gradients can refine. section 3.2

Two-stage training

First, remove a sentence from a passage and train the retriever to recover that passage from the sentence. Then begin joint reader–retriever training, where language-modeling loss further improves document ranking. section 3.2

Figure 3.2 The Two-Stage Training Process of REALM. This diagram illustrates the two distinct phases of training a REALM model.

Use the two-stage training diagram to distinguish inverse-cloze warm-starting from later end-to-end reader–retriever refinement. section 3.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the training rationale and sequence, but stops short of specifying optimization details or implementation code.
Examples
Uses sentence removal and passage recovery as the training example, followed by REALM’s two-stage training process.
What it assumes
Assumes familiarity with retrievers and dual encoders while explaining why random initialization cannot support reader-supervised retrieval.
What you can do after
Build it from scratch by pretraining a retriever on sentence–passage reconstruction before joint reader–retriever optimization.
How it connects
Connects dual encoders, masked language modeling, knowledge retrieval, and end-to-end retriever–reader training.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.