liveBook · Manning

Masked language modeling

As explained in Transfer Learning for Natural Language Processing

Masked language modeling is a training objective that predicts words hidden or masked in an input sequence.

Masked language modeling (MLM) is a fill-in-the-blanks training objective: tokens in training text are randomly masked, and the model predicts them from context. Repeating this task across many examples helps a model learn language representations that can be fine-tuned for downstream NLP tasks. section 8.1 section 1.3.2

BERT uses transformer encoders and self-attention to consider context in both directions for each token. Its training also used next-sentence prediction, a separate objective that asks whether one sentence plausibly follows another. section 8.1

Predicting a hidden word

In “He didn’t want to talk about cells on the cell phone, a [MASK] he considered very boring,” the mask replaces “subject.” A model can rank plausible completions such as “conversation,” “subject,” and “topic”; the surrounding words provide context for the prediction. section 8.1

A pretrained BERT model can also perform fill-in-the-blanks inference without special task-specific fine-tuning. Create a transformers fill-mask pipeline with a model and tokenizer, then pass it a sentence containing [MASK]. section 8.1.3

from transformers import pipeline

fill_mask = pipeline("fill-mask",model="bert-base-cased",tokenizer="bert-base-cased")

Initialize the fill-mask pipeline with bert-base-cased for both the model and tokenizer. section 8.1.3

fill_mask("A new study estimates that if the US had universally mandated masks on 1 April, there could have been nearly 40% fewer [MASK] by the start of June")

Replace a word in a sentence with [MASK] and ask the pipeline for candidate completions. section 8.1.3

For this input, “deaths” is the top suggestion. Other candidates include “executions,” “victims,” and “masks,” illustrating that a plausible completion depends on context and need not reproduce the original word. section 8.1.3

How Transfer Learning for Natural Language Processing covers it

How deep it goes
Explains random masking and bidirectional context, then demonstrates inference without detailing a training implementation.
Examples
Illustrates a masked sentence about a cell phone and ranks completions for a sentence about a public-health study.
What you can do after
Apply it via the transformers fill-mask pipeline to inspect candidate completions.
How it connects
Connects MLM to BERT, transformer encoders, self-attention, transfer learning, and next-sentence prediction.

Also explained in

Compare how 3 books explain masked language modeling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Transfer Learning for Natural Language Processing. The section links point to the parts of the book it draws on.