BERT is a bidirectional Transformer encoder model that builds contextual representations by using both left and right context around each token. Its representations support downstream tasks including sentiment analysis, named entity recognition, and question answering. section 2.3.1
BERT is pretrained with masked language modeling: words are hidden inside a sentence, and the model predicts the missing content from the surrounding context. For example, “The chef cooked [MASK] the meal” requires using information on both sides of the mask rather than predicting only from the preceding words. section 2.3.1
Masked language modeling
Given “The chef cooked [MASK] the meal,” BERT predicts the missing word from the complete surrounding context. This contrasts with left-to-right language modeling, where each word is predicted from preceding words. section 2.3.1
Figure 2.5 The two images depict the difference between language modeling and masked language modeling which gives BERT an advantage over traditional left-to-right sequences.
Compare ordinary left-to-right language modeling with BERT’s masked language modeling objective. section 2.3.1
Figure 2.6 A distinction between Bert (on the left) and Sentence-Bert (on the right) lies in their approach to semantic textual similarity. In BERT, the two sentences are initially concatenated and then fed into the network. Subsequently, the similarity is computed using the activation function. Conversely, in Sentence-Bert, the two sentences are independently input into Bert, and their cosine similarity is then calculated.
Contrast BERT’s concatenated sentence-pair processing with Sentence-BERT’s independent encoding and cosine-similarity calculation. section 2.3.1












