Masked language modeling (MLM) is a fill-in-the-blanks training objective: tokens in training text are randomly masked, and the model predicts them from context. Repeating this task across many examples helps a model learn language representations that can be fine-tuned for downstream NLP tasks. section 8.1 section 1.3.2
BERT uses transformer encoders and self-attention to consider context in both directions for each token. Its training also used next-sentence prediction, a separate objective that asks whether one sentence plausibly follows another. section 8.1
Predicting a hidden word
In “He didn’t want to talk about cells on the cell phone, a [MASK] he considered very boring,” the mask replaces “subject.” A model can rank plausible completions such as “conversation,” “subject,” and “topic”; the surrounding words provide context for the prediction. section 8.1
A pretrained BERT model can also perform fill-in-the-blanks inference without special task-specific fine-tuning. Create a transformers fill-mask pipeline with a model and tokenizer, then pass it a sentence containing [MASK]. section 8.1.3
from transformers import pipeline
fill_mask = pipeline("fill-mask",model="bert-base-cased",tokenizer="bert-base-cased")
Initialize the fill-mask pipeline with bert-base-cased for both the model and tokenizer. section 8.1.3
fill_mask("A new study estimates that if the US had universally mandated masks on 1 April, there could have been nearly 40% fewer [MASK] by the start of June")
Replace a word in a sentence with [MASK] and ask the pipeline for candidate completions. section 8.1.3
For this input, “deaths” is the top suggestion. Other candidates include “executions,” “victims,” and “masks,” illustrating that a plausible completion depends on context and need not reproduce the original word. section 8.1.3


