Masked language modeling

As explained in AI-Powered Search

Masked language modeling is a self-supervised training objective that predicts tokens deliberately removed or masked from an input sequence.

Masked language modeling is a self-supervised objective that removes tokens from an input sequence and trains a model to predict them. BERT applies this Cloze-style task by randomly masking 15% of sentence tokens, using surrounding context through attention. The prediction loss rewards successful reconstruction and produces embeddings that represent words together with their contexts. section 13.3.1

A typical processing sequence is:

• Tokenize text into vocabulary word pieces, including special markers such as [CLS] and [SEP]. section 13.3.1
• Randomly remove 15% of the tokens to create the prediction test. section 13.3.1
• Use surrounding tokens and attention to predict the missing content, optimizing the prediction loss. section 13.3.1

Tokenization example

The term “It’s” can become the word pieces “it”, “'”, and “s” before entering the masked-language-modeling pipeline. section 13.3.1

Figure 13.4 The Transformer encoder

Trace the Transformer encoder that converts tokenized text into contextual representations. section 13.3.1

How AI-Powered Search covers it

How deep it goes

Explains masking, attention, loss optimization, tokenization, and contextual embeddings, but stops short of implementation code.

Examples

Uses random masking of sentence tokens and splits “It’s” into three word pieces.

What it assumes

Assumes basic embeddings and introduces the Cloze-style prediction task, attention, and word-piece tokenization.

What you can do after

Understand how masked prediction produces contextual embeddings from tokenized text.

How it connects

Connects the Cloze test, self-supervised learning, attention mechanisms, Transformers, BERT, tokenization, and embeddings.

liveBookmanning.com

Read AI-Powered Search on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionMasked language modeling is a pretraining technique that masks tokens in an input sequence and trains a model to predict the original tokens from surrounding context.

Compare how 2 books explain masked language modeling →

Related concepts

• BERTused by · BERT is a pretrained Transformer encoder language model that uses bidirectional context to produce contextual representations of text.
• causal language modelcontrasts with · A causal language model predicts each token from preceding tokens and generates sequences autoregressively.
• encoder-only transformerused by · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.

This explainer was generated by Manning's knowledge graph from the text of AI-Powered Search. The section links point to the parts of the book it draws on.