liveBook · Manning

BERT

As explained in Retrieval Augmented Generation, The Foundational Ideas

BERT is a bidirectional Transformer-based language model pretrained to represent text using contextual token representations.

A support ticket about a connection that keeps dropping should match earlier tickets about network stability, even when they use different words. BERT provides contextual representations that help capture meaning beyond a simple match between words. Its usefulness depends on where it sits in the search system: reading a few candidate passages is different from comparing a query with every stored item. section 2.3 section 3.1

BERT stands for Bidirectional Encoder Representations from Transformers. Unlike a fixed Word2Vec vector, its representation of “bank” can change when the surrounding words suggest a river rather than money. Transformer self-attention relates words across a sentence, while BERT’s bidirectional processing uses context on both sides of a word. Its understanding was learned through a fill-in-the-blank training task over billions of words. section 2.3 section 3.1

Two uses expose different constraints:

  • As a fine-tuned question-answering reader, BERT examines passages selected by a retriever and finds an answer span. It cannot answer from a relevant passage that keyword retrieval never supplies. section 3.1
  • For sentence similarity, applying BERT to every text pair is expensive: comparing one new ticket with 100,000 resolved tickets requires 100,000 forward passes. Sentence-BERT addresses the need for practical large-scale similarity search. section 2.3

When a capable reader never sees the answer

An employee asks how much they can put into a 401(k), while the relevant document states a “maximum employee deferral.” A keyword-based retriever may miss that passage. BERT may be able to identify the answer within it, but only if retrieval brings the passage to the reader. section 3.1

Choose the role before choosing the model. BERT is useful for contextual reading of retrieved passages; using it to score every possible pair can make similarity search impractical. In one reported comparison, finding the most similar pair among 10,000 sentences required about 50 million BERT inferences—roughly 65 hours on a V100. section 2.3 section 3.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains bidirectional context, self-attention, pretraining, answer-span reading, and pairwise inference cost without detailing BERT implementation.
Examples
Uses ambiguous “bank,” support-ticket matching, a 10,000-sentence cost measurement, and a missed 401(k) answer passage.
What it assumes
Builds from contrasts with fixed word vectors, one-directional reading, and keyword retrieval rather than assuming their mechanics.
What you can do after
Evaluate/understand when contextual reading helps and when pairwise cost or failed retrieval limits BERT.
How it connects
Connects BERT to Transformers, self-attention, Word2Vec, Sentence-BERT, sparse retrieval, and open-domain question answering.

Also explained in

Compare how 13 books explain BERT →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.