A contextualized representation is a numerical embedding whose value reflects surrounding sequence context. BERT generates these representations by using information from both the left and right sides of a token, rather than processing text in only one direction. section 2.3.1
The representation pipeline connects several modeling choices:
- Masked language modeling trains BERT to predict missing words from their surrounding context. section 2.3.1
- The resulting embeddings support sentiment analysis, named entity recognition, and question answering. section 2.3.1
- Sentence-BERT extends the idea from individual words to sentence embeddings optimized for semantic textual similarity. section 2.3.1
Masked context
Given “The chef cooked [MASK] the meal,” masked language modeling uses the surrounding words to predict the missing token, “the.” section 2.3.1
Figure 2.5 The two images depict the difference between language modeling and masked language modeling which gives BERT an advantage over traditional left-to-right sequences.
Compare left-to-right language modeling with BERT’s bidirectional masked language modeling. section 2.3.1






