liveBook · Manning

Masked language modeling

As explained in Retrieval Augmented Generation, The Foundational Ideas

Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden from an input sequence.

Masked language modeling (MLM) trains a model to recover hidden text. In a BERT-style task, words in real sentences are replaced with [MASK]; the original words become the answers, so no external labels are needed. Retrieval-augmented models can use prediction performance to train a retriever as well as a reader: a useful passage is one that makes the missing text easier to predict. section 3.2 section 3.5

A blank that needs evidence

Given “The monarch butterfly migrates to [MASK] for the winter,” a retriever can fetch a passage mentioning migration to Mexico. The reader uses that passage to predict “Mexico.” Masking a routine function word would be less useful for training retrieval: grammar alone might supply the answer. REALM therefore emphasizes salient spans such as named entities and dates. section 3.2

Atlas uses a related prefix-MLM objective: it corrupts text, asks the model to reconstruct it, and judges retrieved passages by whether they help. It excludes the passage being reconstructed from retrieval. Otherwise, the retriever could fetch the answer itself, lowering the loss without learning to find useful evidence. Keeping the retrieval index current also adds pretraining cost. section 6.2

Table 3.2 Differences between SBERT, DPR, and REALM
  Model  
  Training Objective  
  Supervision Signal  
  Resulting Vector Space Property  
 SBERT 
 Cosine Similarity / Contrastive Loss on semantic labels 
 Human-annotated sentence pairs 
 Optimized for broad semantic similarity and paraphrase detection 
 DPR 
 Contrastive Loss on QA pairs (Negative Log-Likelihood) 
 Labeled (question, answer_passage) pairs 
 Asymmetric space optimized for question-to-answer-passage relevance 
 REALM 
 Maximize Marginal Log-Likelihood of Masked Language Modeling 
 Downstream MLM task performance 
 Optimized for instrumental value to a language model. 

Compare REALM’s prediction-based retrieval objective with SBERT’s semantic-pair supervision and DPR’s labeled question–passage pairs. section 3.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains masking and retrieval feedback through a worked prediction, then identifies salient masking and answer-leakage safeguards without giving implementation code.
Examples
Uses a masked monarch-migration sentence, a Mexico passage, and Atlas’s exclusion of the passage being reconstructed.
What you can do after
Evaluate/understand whether a masking choice rewards evidence retrieval or lets the model predict from grammar or a leaked answer.
How it connects
Connects self-supervised learning and salient span masking to reader–retriever training and retrieval-index staleness.

Also explained in

Compare how 6 books explain masked language modeling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.