A randomly initialized retriever fetches unhelpful documents, leaving reader supervision without a useful signal. The Inverse Cloze Task warm-starts retrieval before joint training by matching a sentence with the passage from which it was removed. section 3.2
The task trains a dual-encoder retriever to place a sentence and its source passage close in embedding space. This gives retrieval a partial capability that downstream masked-language-modeling gradients can refine. section 3.2
Two-stage training
First, remove a sentence from a passage and train the retriever to recover that passage from the sentence. Then begin joint reader–retriever training, where language-modeling loss further improves document ranking. section 3.2
Figure 3.2 The Two-Stage Training Process of REALM. This diagram illustrates the two distinct phases of training a REALM model.

Use the two-stage training diagram to distinguish inverse-cloze warm-starting from later end-to-end reader–retriever refinement. section 3.2
