Masked language modeling
Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.
Build AI Drug Discovery Pipelines
Masked language modeling is a self-supervised technique that trains a model to predict tokens hidden or replaced within an input sequence.
- How it is used
- Uses self-supervised pretraining to learn protein representations for downstream drug-discovery tasks.
- How deep it goes
- Explains masked-residue prediction and implements a small transformer model, then relates the approach to scaled ESM-2 models.
- Examples
- Uses a masked lysine prediction and a small protein language model, alongside ESM-2 and antibody classification.
- What you can do after
- Builds a small protein language model and evaluates masked-token predictions on held-out sequences.
Read how Build AI Drug Discovery Pipelines explains masked language modeling →
Retrieval Augmented Generation, The Foundational Ideas
Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden from an input sequence.
- How it is used
- Supplies a self-supervised prediction signal that trains retrieval toward passages useful to a language-model reader.
- How deep it goes
- Explains masking and retrieval feedback through a worked prediction, then identifies salient masking and answer-leakage safeguards without giving implementation code.
- Examples
- Uses a masked monarch-migration sentence, a Mexico passage, and Atlas’s exclusion of the passage being reconstructed.
- What you can do after
- Evaluate/understand whether a masking choice rewards evidence retrieval or lets the model predict from grammar or a leaked answer.
Read how Retrieval Augmented Generation, The Foundational Ideas explains masked language modeling →
Natural Language Processing in Action, Second Edition
Masked language modeling is a pretraining technique that masks tokens in an input sequence and trains a model to predict the original tokens from surrounding context.
- How it is used
- Trains bidirectional transformer representations for later sentence-level and token-level language tasks.
- How deep it goes
- Explains masking rates, replacement probabilities, token prediction, and the distinction from causal training.
- Examples
- Uses masked WordPiece tokens, BERT pretraining on Wikipedia and BooksCorpus, and Hugging Face masking configuration as concrete examples.
- What you can do after
- Apply it via a Hugging Face data collator by enabling random masking for bidirectional language-model training.
Read how Natural Language Processing in Action, Second Edition explains masked language modeling →
Transfer Learning for Natural Language Processing
Masked language modeling is a training objective that predicts words hidden or masked in an input sequence.
- How it is used
- Uses masked-token prediction to train BERT’s language representations for transfer to downstream NLP tasks.
- How deep it goes
- Explains random masking and bidirectional context, then demonstrates inference without detailing a training implementation.
- Examples
- Illustrates a masked sentence about a cell phone and ranks completions for a sentence about a public-health study.
- What you can do after
- Apply it via the transformers fill-mask pipeline to inspect candidate completions.
Read how Transfer Learning for Natural Language Processing explains masked language modeling →
AI-Powered Search
Masked language modeling is a self-supervised training objective that predicts tokens deliberately removed or masked from an input sequence.
- How it is used
- Uses unlabeled text to pretrain contextual embeddings for semantic search and language understanding.
- How deep it goes
- Explains masking, attention, loss optimization, tokenization, and contextual embeddings, but stops short of implementation code.
- Examples
- Uses random masking of sentence tokens and splits “It’s” into three word pieces.
- What you can do after
- Understand how masked prediction produces contextual embeddings from tokenized text.
Read how AI-Powered Search explains masked language modeling →
Build an Advanced RAG Application (From Scratch)
Masked language modeling is a pretraining technique that predicts tokens hidden or replaced by mask markers using surrounding sequence context.
- How it is used
- Uses masked token prediction to pretrain encoder models for richer language representations and downstream NLP tasks.
- How deep it goes
- Explains the bidirectional prediction objective and illustrates it with a masked sentence, without detailing training implementation.
- Examples
- Uses “The chef cooked [MASK] the meal” to demonstrate predicting a missing token from surrounding context.
- What you can do after
- Understand how BERT uses bidirectional masked prediction to learn contextual representations.
Read how Build an Advanced RAG Application (From Scratch) explains masked language modeling →
Also covered in
- Deep Learning with Python, Third Edition
- Deep Learning with R, Third Edition
- Transformers in Action
- Hugging Face in Action
- Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
What to know first
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- language modelingLanguage modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context.
Related concepts
- BERTused by · BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- causal language modelcontrasts with · A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation.
- encoder-only transformerused by · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
- REALM++used by · REALM++ is a method for improving REALM through changes to training scale, retrieval during evaluation, and supervision without changing its model architecture.
Questions about masked language modeling
What is masked language modeling?
Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.
Which Manning books explain masked language modeling?
Build AI Drug Discovery Pipelines, Retrieval Augmented Generation, The Foundational Ideas, Natural Language Processing in Action, Second Edition, Transfer Learning for Natural Language Processing, AI-Powered Search and Build an Advanced RAG Application (From Scratch).
What should I know before learning masked language modeling?
Manning books that teach masked language modeling usually introduce transformer architecture, embedding, attention and language modeling first.
What is the difference between masked language modeling and causal language model?
Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context. A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation.
What kinds of masked language modeling are there?
Kinds of masked language modeling include prefix-mlm.
What is masked language modeling also called?
Masked language modeling is also called MLM.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





