liveBook · Manning

Masked language modeling

As explained in Build an Advanced RAG Application (From Scratch)

Masked language modeling is a pretraining technique that predicts tokens hidden or replaced by mask markers using surrounding sequence context.

Masked language modeling (MLM) pretrains encoder models by hiding a token and predicting it from both left and right sequence context. This bidirectional objective helps produce contextual embeddings for downstream NLP tasks. section 2.3.1

Toy prediction

Given “The chef cooked [MASK] the meal,” the model predicts the missing token from the surrounding sentence, such as “the.” section 2.3.1

Figure 2.5 The two images depict the difference between language modeling and masked language modeling which gives BERT an advantage over traditional left-to-right sequences.

Compare ordinary left-to-right language modeling with bidirectional masked prediction. section 2.3.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains the bidirectional prediction objective and illustrates it with a masked sentence, without detailing training implementation.
Examples
Uses “The chef cooked [MASK] the meal” to demonstrate predicting a missing token from surrounding context.
What you can do after
Understand how BERT uses bidirectional masked prediction to learn contextual representations.
How it connects
Connects masked language modeling to BERT, bidirectional context, contextual embeddings, and downstream NLP tasks.

Also explained in

Compare how 6 books explain masked language modeling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.