liveBook · Manning

BERT

As explained in LLM Evaluation and Alignment, The Foundational Ideas

BERT is a bidirectional Transformer encoder model pretrained to produce contextual representations for natural-language understanding.

BERT (Bidirectional Encoder Representations from Transformers) is a Transformer encoder that produces context-dependent representations for natural-language understanding. Unlike static word vectors, it gives the same word different representations in different contexts and preserves relationships between words. section 3.2.2

BERT’s practical value comes from how it represents text:

  • Reads context bidirectionally, allowing a token’s representation to use words on both sides rather than only preceding tokens. section 3.2.2
  • Distinguishes meanings such as financial “bank” and riverbank “bank” from surrounding words. section 3.2.2
  • Supports semantic comparison by providing contextual token or sentence representations for metrics such as BERTScore. section 3.2.2 section 3.5

Encoding two equivalent passages

Load a pretrained BERT tokenizer and encoder, tokenize each passage with truncation and padding, run inference without gradient tracking, extract a representation, and compare the passages with cosine similarity. Mean pooling aggregates token embeddings while masking padded positions; the [CLS] token offers another whole-input representation. section 3.2.2

Listing 3.1 Preparing a BERT model for encoding .
python
from transformers import BertTokenizer, BertModel
import torch
 
tokenizer = BertTokenizer.from_pretrained(‘bert-base-uncased’)
encoder = BertModel.from_pretrained(‘bert-base-uncased’)
encoder.eval()                                                  #A

Inspect the Python setup for loading a pretrained bert-base-uncased tokenizer and encoder. section 3.2.2

Listing 3.4 Encoding text inputs into embeddings using mean pooling .
python
def get_embeddings(text: str):
    inputs = tokenizer(text, return_tensors=’pt’, truncation=True, padding=True)
    with torch.no_grad():
        Outputs = model(**inputs)
    
    token_embeddings = outputs.last_hidden_state                          #A
    attention_mask = encoded["attention_mask"]                            #B
 
    #C
    mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    masked_embeddings = token_embeddings * mask_expanded
 
    #D
    sum_embeddings = masked_embeddings.sum(dim=1)
    token_counts = mask_expanded.sum(dim=1).clamp(min=1e-9)
 
    return = sum_embeddings / token_counts

Inspect mean pooling with an attention mask to aggregate token embeddings while excluding padding. section 3.2.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains contextual meaning, bidirectional attention, pooling, and cosine comparison, but defers BERT’s training procedure and architecture internals.
Examples
Uses bank disambiguation, agent-patient word order, legal-context encoding, pooling, and cosine similarity between paraphrased passages.
What it assumes
Assumes basic text representations and similarity concepts while contrasting contextual embeddings with static word vectors.
What you can do after
Apply it via Transformers to encode text and compare contextual representations with cosine similarity or BERTScore.
How it connects
Connects Transformer encoders with contextual embeddings, BERTScore, COMET, static word vectors, and autoregressive decoders.

Also explained in

Compare how 13 books explain BERT →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.