liveBook · Manning

BERT

As explained in AI-Powered Search

BERT is a transformer-based language model pretrained to produce contextual representations of text using bidirectional attention.

BERT is a Transformer-based language model that learns contextual representations by attending to surrounding tokens in both directions. It became notable for strong language-comprehension performance and for learning from unlabeled text through self-supervision. section 13.3.1

During pretraining, BERT creates Cloze-style tasks by randomly masking about 15% of sentence tokens. It predicts the missing tokens from their surrounding context, optimizing a loss function so its output vectors encode both word meaning and context. section 13.3.1

A typical contextual-embedding workflow proceeds as follows:

  • Start with token-level embeddings and expose the model to masked sentences. section 13.3.1
  • Use bidirectional attention to incorporate information from surrounding tokens. section 13.3.1
  • Reuse the resulting vectors for semantic comparison or downstream language tasks. section 13.3.1 section 13.4.3

From BERT embeddings to sentence similarity

SBERT adapts the BERT idea for sentence-level comparison by pooling token embeddings into one vector and fine-tuning similarity behavior. A SentenceTransformer workflow loads a pretrained model, encodes phrases such as "it's raining hard" and "it is wet outside," normalizes the resulting 768-dimensional vectors, and compares them with dot products. The highest similarity pairs the two weather-related phrases, ahead of unrelated vehicle phrases. section 13.4.3

Figure 13.4 The Transformer encoder
figure

Use the Transformer encoder figure to inspect the attention-based architecture underlying contextual token representations. section 13.3.1

How AI-Powered Search covers it

How deep it goes
Explains masked-token pretraining, attention, embeddings, and sentence pooling, but defers full Transformer implementation details.
Examples
Uses masked sentences, weather phrases, vehicle phrases, 768-dimensional vectors, and a similarity matrix as teaching artifacts.
What it assumes
Assumes basic familiarity with embeddings and language models while explaining attention, masking, and self-supervised training.
What you can do after
Apply it via SentenceTransformer to encode phrases, normalize embeddings, and rank pairs by similarity.
How it connects
Connects BERT to Transformers, attention, masked language modeling, embeddings, semantic search, and SBERT.

Also explained in

Compare how 13 books explain BERT →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI-Powered Search. The section links point to the parts of the book it draws on.