Masked language modeling (MLM) trains a model to reconstruct hidden tokens using context on both sides. Instead of predicting only the next token, it randomly masks tokens and predicts their original identities, helping a bidirectional transformer learn richer language representations. section 9.3
A typical MLM pretraining procedure works as follows:
- Select 15% of the WordPiece tokens for prediction. section 9.3.1
- Replace selected tokens with [MASK] 80% of the time, random tokens 10% of the time, and the original tokens 10% of the time. section 9.3.1
- Predict the original token IDs from the surrounding representation, formed from token, segment, and positional embeddings. section 9.3.1
- Optionally combine MLM with next sentence prediction when downstream tasks require relationships between sentence pairs. section 9.3.1
BERT-style pretraining
BERT uses stacked encoder transformers with feedforward and attention layers. It can be pretrained on large corpora such as English Wikipedia and BooksCorpus, then fine-tuned—with changes to inputs and the output layer—for sentence-level and token-level tasks such as question answering. section 9.3 section 9.3.1
MLM is acausal: it can use both left and right context, but it is not suited to ordinary left-to-right text generation. In Hugging Face training pipelines, random masking requires DataCollatorForLanguageModeling with mlm=True; mlm=False selects causal training for models such as GPT-2. section 10.2.2
