liveBook · Manning

Masked language modeling

Also called MLM

Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.

How 6 Manning books explain it

Build AI Drug Discovery Pipelines

Masked language modeling is a self-supervised technique that trains a model to predict tokens hidden or replaced within an input sequence.

How it is used
Uses self-supervised pretraining to learn protein representations for downstream drug-discovery tasks.
How deep it goes
Explains masked-residue prediction and implements a small transformer model, then relates the approach to scaled ESM-2 models.
Examples
Uses a masked lysine prediction and a small protein language model, alongside ESM-2 and antibody classification.
What you can do after
Builds a small protein language model and evaluates masked-token predictions on held-out sequences.

Read how Build AI Drug Discovery Pipelines explains masked language modeling →

Retrieval Augmented Generation, The Foundational Ideas

Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden from an input sequence.

How it is used
Supplies a self-supervised prediction signal that trains retrieval toward passages useful to a language-model reader.
How deep it goes
Explains masking and retrieval feedback through a worked prediction, then identifies salient masking and answer-leakage safeguards without giving implementation code.
Examples
Uses a masked monarch-migration sentence, a Mexico passage, and Atlas’s exclusion of the passage being reconstructed.
What you can do after
Evaluate/understand whether a masking choice rewards evidence retrieval or lets the model predict from grammar or a leaked answer.

Read how Retrieval Augmented Generation, The Foundational Ideas explains masked language modeling →

Natural Language Processing in Action, Second Edition

Masked language modeling is a pretraining technique that masks tokens in an input sequence and trains a model to predict the original tokens from surrounding context.

How it is used
Trains bidirectional transformer representations for later sentence-level and token-level language tasks.
How deep it goes
Explains masking rates, replacement probabilities, token prediction, and the distinction from causal training.
Examples
Uses masked WordPiece tokens, BERT pretraining on Wikipedia and BooksCorpus, and Hugging Face masking configuration as concrete examples.
What you can do after
Apply it via a Hugging Face data collator by enabling random masking for bidirectional language-model training.

Read how Natural Language Processing in Action, Second Edition explains masked language modeling →

Transfer Learning for Natural Language Processing

Masked language modeling is a training objective that predicts words hidden or masked in an input sequence.

How it is used
Uses masked-token prediction to train BERT’s language representations for transfer to downstream NLP tasks.
How deep it goes
Explains random masking and bidirectional context, then demonstrates inference without detailing a training implementation.
Examples
Illustrates a masked sentence about a cell phone and ranks completions for a sentence about a public-health study.
What you can do after
Apply it via the transformers fill-mask pipeline to inspect candidate completions.

Read how Transfer Learning for Natural Language Processing explains masked language modeling →

AI-Powered Search

Masked language modeling is a self-supervised training objective that predicts tokens deliberately removed or masked from an input sequence.

How it is used
Uses unlabeled text to pretrain contextual embeddings for semantic search and language understanding.
How deep it goes
Explains masking, attention, loss optimization, tokenization, and contextual embeddings, but stops short of implementation code.
Examples
Uses random masking of sentence tokens and splits “It’s” into three word pieces.
What you can do after
Understand how masked prediction produces contextual embeddings from tokenized text.

Read how AI-Powered Search explains masked language modeling →

Build an Advanced RAG Application (From Scratch)

Masked language modeling is a pretraining technique that predicts tokens hidden or replaced by mask markers using surrounding sequence context.

How it is used
Uses masked token prediction to pretrain encoder models for richer language representations and downstream NLP tasks.
How deep it goes
Explains the bidirectional prediction objective and illustrates it with a masked sentence, without detailing training implementation.
Examples
Uses “The chef cooked [MASK] the meal” to demonstrate predicting a missing token from surrounding context.
What you can do after
Understand how BERT uses bidirectional masked prediction to learn contextual representations.

Read how Build an Advanced RAG Application (From Scratch) explains masked language modeling →

Also covered in

What to know first

Related concepts

Questions about masked language modeling

What is masked language modeling?

Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.

Which Manning books explain masked language modeling?

Build AI Drug Discovery Pipelines, Retrieval Augmented Generation, The Foundational Ideas, Natural Language Processing in Action, Second Edition, Transfer Learning for Natural Language Processing, AI-Powered Search and Build an Advanced RAG Application (From Scratch).

What should I know before learning masked language modeling?

Manning books that teach masked language modeling usually introduce transformer architecture, embedding, attention and language modeling first.

What is the difference between masked language modeling and causal language model?

Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context. A causal language model predicts each token from preceding tokens only, without access to future tokens, enabling autoregressive sequence generation.

What kinds of masked language modeling are there?

Kinds of masked language modeling include prefix-mlm.

What is masked language modeling also called?

Masked language modeling is also called MLM.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.