liveBook · Manning

Contextual embedding

As explained in Build an Advanced RAG Application (From Scratch)

A contextual embedding is a numerical representation of a token whose value reflects the token's surrounding sequence context.

A contextualized representation is a numerical embedding whose value reflects surrounding sequence context. BERT generates these representations by using information from both the left and right sides of a token, rather than processing text in only one direction. section 2.3.1

The representation pipeline connects several modeling choices:

  • Masked language modeling trains BERT to predict missing words from their surrounding context. section 2.3.1
  • The resulting embeddings support sentiment analysis, named entity recognition, and question answering. section 2.3.1
  • Sentence-BERT extends the idea from individual words to sentence embeddings optimized for semantic textual similarity. section 2.3.1

Masked context

Given “The chef cooked [MASK] the meal,” masked language modeling uses the surrounding words to predict the missing token, “the.” section 2.3.1

Figure 2.5 The two images depict the difference between language modeling and masked language modeling which gives BERT an advantage over traditional left-to-right sequences.

Compare left-to-right language modeling with BERT’s bidirectional masked language modeling. section 2.3.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains bidirectional context, masked language modeling, and sentence-level extension, but does not detail embedding dimensions or implementation code.
Examples
Uses a masked chef-and-meal sentence and downstream tasks including sentiment analysis, named entity recognition, and question answering.
What you can do after
Evaluate/understand how bidirectional context produces token representations and how SBERT adapts them to sentence similarity.
How it connects
Connects Transformer models, BERT, masked language modeling, Sentence-BERT, sentence embeddings, and semantic textual similarity.

Also explained in

Compare how 7 books explain contextual embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.