Masked language modeling (MLM) trains a model to recover hidden text. In a BERT-style task, words in real sentences are replaced with [MASK]; the original words become the answers, so no external labels are needed. Retrieval-augmented models can use prediction performance to train a retriever as well as a reader: a useful passage is one that makes the missing text easier to predict. section 3.2 section 3.5
A blank that needs evidence
Given “The monarch butterfly migrates to [MASK] for the winter,” a retriever can fetch a passage mentioning migration to Mexico. The reader uses that passage to predict “Mexico.” Masking a routine function word would be less useful for training retrieval: grammar alone might supply the answer. REALM therefore emphasizes salient spans such as named entities and dates. section 3.2
Atlas uses a related prefix-MLM objective: it corrupts text, asks the model to reconstruct it, and judges retrieved passages by whether they help. It excludes the passage being reconstructed from retrieval. Otherwise, the retriever could fetch the answer itself, lowering the loss without learning to find useful evidence. Keeping the retrieval index current also adds pretraining cost. section 6.2
Table 3.2 Differences between SBERT, DPR, and REALM
|
Model
|
Training
Objective
|
Supervision
Signal
|
Resulting Vector
Space Property
|
|
SBERT
|
Cosine Similarity /
Contrastive Loss on semantic labels
|
Human-annotated
sentence pairs
|
Optimized for broad
semantic similarity and paraphrase detection
|
|
DPR
|
Contrastive Loss on
QA pairs (Negative Log-Likelihood)
|
Labeled (question,
answer_passage) pairs
|
Asymmetric space
optimized for question-to-answer-passage relevance
|
|
REALM
|
Maximize Marginal
Log-Likelihood of Masked Language Modeling
|
Downstream MLM task
performance
|
Optimized for
instrumental value to a language model.
|
Compare REALM’s prediction-based retrieval objective with SBERT’s semantic-pair supervision and DPR’s labeled question–passage pairs. section 3.2





