liveBook · Manning

Contextual embedding

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A contextual embedding is a vector representation of a token whose value reflects the token's surrounding linguistic context.

A contextualized representation, or contextual embedding, is a vector for a token whose value reflects its surrounding words. It helps compare meaning when two texts use different wording, where lexical overlap alone can miss a paraphrase. section 3.2.2 section 2.3.6

One word, different contexts

In a sentence about depositing money, the embedding for “bank” reflects a financial institution. In a sentence about a river and a picnic, it reflects a riverbank. A static word vector would give “bank” the same representation in both contexts. section 3.2.2 section 3.5

BERT produces context-dependent vectors using self-attention, which also encodes relationships between words. For example, “dog” plays a different role in “The dog bit the man” than in “The man bit the dog.” Similar meanings can occupy nearby regions of vector space; cosine similarity compares vector orientation rather than magnitude. section 3.2.2 section 3.2.3

Figure 3.3 Encoders like BERT read bidirectionally. When processing “attorney”, they can see “filed” and “brief” to understand its legal context. Decoders like GPT read left-to-right for token generation, unable to use future context.

The diagram illustrates how a bidirectional encoder uses words on both sides of a token to interpret its context. section 3.2.2

BERTScore uses contextual token embeddings to compare a candidate with a reference. It greedily finds a semantic match for each token, so related terms such as “attorney” and “lawyer” can contribute even without an exact word match. BERT is a model; BERTScore is an evaluation algorithm that uses encoder models. section 3.2.2 section 3.2.5 section 3.5

Semantic similarity is not a complete quality assessment. BERTScore requires a reference and can miss the distinction between a minor stylistic difference and a critical factual error. Semantic metrics also cost more to compute, are harder to interpret, and can inherit language-model biases. section 3.2.5 section 2.3.6

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains context-sensitive vectors, self-attention, geometric similarity, and greedy token matching, while distinguishing semantic similarity from overall quality.
Examples
Contrasts financial and river senses of “bank,” reversed biting roles, and matching “attorney” with “lawyer.”
What you can do after
Evaluate/understand how contextual embeddings support semantic matching and why similarity scores alone cannot establish output quality.
How it connects
Connects contextual embeddings to static word embeddings, self-attention, cosine similarity, and BERTScore.

Also explained in

Compare how 7 books explain contextual embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.