liveBook · Manning

BERT

As explained in Build an Advanced RAG Application (From Scratch)

BERT is a bidirectional Transformer-based language model that produces contextual representations by attending to both sides of input tokens.

BERT is a bidirectional Transformer encoder model that builds contextual representations by using both left and right context around each token. Its representations support downstream tasks including sentiment analysis, named entity recognition, and question answering. section 2.3.1

BERT is pretrained with masked language modeling: words are hidden inside a sentence, and the model predicts the missing content from the surrounding context. For example, “The chef cooked [MASK] the meal” requires using information on both sides of the mask rather than predicting only from the preceding words. section 2.3.1

Masked language modeling

Given “The chef cooked [MASK] the meal,” BERT predicts the missing word from the complete surrounding context. This contrasts with left-to-right language modeling, where each word is predicted from preceding words. section 2.3.1

Figure 2.5 The two images depict the difference between language modeling and masked language modeling which gives BERT an advantage over traditional left-to-right sequences.

Compare ordinary left-to-right language modeling with BERT’s masked language modeling objective. section 2.3.1

Figure 2.6 A distinction between Bert (on the left) and Sentence-Bert (on the right) lies in their approach to semantic textual similarity. In BERT, the two sentences are initially concatenated and then fed into the network. Subsequently, the similarity is computed using the activation function. Conversely, in Sentence-Bert, the two sentences are independently input into Bert, and their cosine similarity is then calculated.

Contrast BERT’s concatenated sentence-pair processing with Sentence-BERT’s independent encoding and cosine-similarity calculation. section 2.3.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains bidirectional context, masked language modeling, contextual embeddings, and sentence-pair differences, without implementation details.
Examples
Uses “The chef cooked [MASK] the meal” to contrast masked prediction with left-to-right language modeling.
What it assumes
Distinguishes traditional language modeling from Transformer encoder processing and masked language modeling.
What you can do after
Apply contextual representations to sentiment analysis, named entity recognition, and question answering.
How it connects
Connects BERT to Transformer encoders, masked language modeling, contextual embeddings, Sentence-BERT, and semantic textual similarity.

Also explained in

Compare how 13 books explain BERT →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.