BERT is a Transformer-based language model that learns contextual representations by attending to surrounding tokens in both directions. It became notable for strong language-comprehension performance and for learning from unlabeled text through self-supervision. section 13.3.1
During pretraining, BERT creates Cloze-style tasks by randomly masking about 15% of sentence tokens. It predicts the missing tokens from their surrounding context, optimizing a loss function so its output vectors encode both word meaning and context. section 13.3.1
A typical contextual-embedding workflow proceeds as follows:
- Start with token-level embeddings and expose the model to masked sentences. section 13.3.1
- Use bidirectional attention to incorporate information from surrounding tokens. section 13.3.1
- Reuse the resulting vectors for semantic comparison or downstream language tasks. section 13.3.1 section 13.4.3
From BERT embeddings to sentence similarity
SBERT adapts the BERT idea for sentence-level comparison by pooling token embeddings into one vector and fine-tuning similarity behavior. A SentenceTransformer workflow loads a pretrained model, encodes phrases such as "it's raining hard" and "it is wet outside," normalizes the resulting 768-dimensional vectors, and compares them with dot products. The highest similarity pairs the two weather-related phrases, ahead of unrelated vehicle phrases. section 13.4.3
Figure 13.4 The Transformer encoder
Use the Transformer encoder figure to inspect the attention-based architecture underlying contextual token representations. section 13.3.1












