Masked language modeling
Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
Natural Language Processing in Action, Second Edition
Masked language modeling is a pretraining technique that masks tokens in an input sequence and trains a model to predict the original tokens from surrounding context.
- How it is used
- Trains bidirectional transformer representations for later sentence-level and token-level language tasks.
- How deep it goes
- Explains masking rates, replacement probabilities, token prediction, and the distinction from causal training.
- Examples
- Uses masked WordPiece tokens, BERT pretraining on Wikipedia and BooksCorpus, and Hugging Face masking configuration as concrete examples.
- What you can do after
- Apply it via a Hugging Face data collator by enabling random masking for bidirectional language-model training.
Read how Natural Language Processing in Action, Second Edition explains masked language modeling →
AI-Powered Search
Masked language modeling is a self-supervised training objective that predicts tokens deliberately removed or masked from an input sequence.
- How it is used
- Uses unlabeled text to pretrain contextual embeddings for semantic search and language understanding.
- How deep it goes
- Explains masking, attention, loss optimization, tokenization, and contextual embeddings, but stops short of implementation code.
- Examples
- Uses random masking of sentence tokens and splits “It’s” into three word pieces.
- What you can do after
- Understand how masked prediction produces contextual embeddings from tokenized text.
Read how AI-Powered Search explains masked language modeling →
Related concepts
- BERTused by · BERT is a pretrained Transformer encoder language model that uses bidirectional context to produce contextual representations of text.
- causal language modelcontrasts with · A causal language model predicts each token from preceding tokens and generates sequences autoregressively.
- encoder-only transformerused by · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.