BERT (Bidirectional Encoder Representations from Transformers) is a Transformer encoder that produces context-dependent representations for natural-language understanding. Unlike static word vectors, it gives the same word different representations in different contexts and preserves relationships between words. section 3.2.2
BERT’s practical value comes from how it represents text:
- Reads context bidirectionally, allowing a token’s representation to use words on both sides rather than only preceding tokens. section 3.2.2
- Distinguishes meanings such as financial “bank” and riverbank “bank” from surrounding words. section 3.2.2
- Supports semantic comparison by providing contextual token or sentence representations for metrics such as BERTScore. section 3.2.2 section 3.5
Encoding two equivalent passages
Load a pretrained BERT tokenizer and encoder, tokenize each passage with truncation and padding, run inference without gradient tracking, extract a representation, and compare the passages with cosine similarity. Mean pooling aggregates token embeddings while masking padded positions; the [CLS] token offers another whole-input representation. section 3.2.2
Listing 3.1 Preparing a BERT model for encoding .
python from transformers import BertTokenizer, BertModel import torch tokenizer = BertTokenizer.from_pretrained(‘bert-base-uncased’) encoder = BertModel.from_pretrained(‘bert-base-uncased’) encoder.eval() #A
Inspect the Python setup for loading a pretrained bert-base-uncased tokenizer and encoder. section 3.2.2
Listing 3.4 Encoding text inputs into embeddings using mean pooling .
python
def get_embeddings(text: str):
inputs = tokenizer(text, return_tensors=’pt’, truncation=True, padding=True)
with torch.no_grad():
Outputs = model(**inputs)
token_embeddings = outputs.last_hidden_state #A
attention_mask = encoded["attention_mask"] #B
#C
mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
masked_embeddings = token_embeddings * mask_expanded
#D
sum_embeddings = masked_embeddings.sum(dim=1)
token_counts = mask_expanded.sum(dim=1).clamp(min=1e-9)
return = sum_embeddings / token_counts
Inspect mean pooling with an attention mask to aggregate token embeddings while excluding padding. section 3.2.2












