A contextualized representation, or contextual embedding, is a vector for a token whose value reflects its surrounding words. It helps compare meaning when two texts use different wording, where lexical overlap alone can miss a paraphrase. section 3.2.2 section 2.3.6
One word, different contexts
In a sentence about depositing money, the embedding for “bank” reflects a financial institution. In a sentence about a river and a picnic, it reflects a riverbank. A static word vector would give “bank” the same representation in both contexts. section 3.2.2 section 3.5
BERT produces context-dependent vectors using self-attention, which also encodes relationships between words. For example, “dog” plays a different role in “The dog bit the man” than in “The man bit the dog.” Similar meanings can occupy nearby regions of vector space; cosine similarity compares vector orientation rather than magnitude. section 3.2.2 section 3.2.3
Figure 3.3 Encoders like BERT read bidirectionally. When processing “attorney”, they can see “filed” and “brief” to understand its legal context. Decoders like GPT read left-to-right for token generation, unable to use future context.
The diagram illustrates how a bidirectional encoder uses words on both sides of a token to interpret its context. section 3.2.2
BERTScore uses contextual token embeddings to compare a candidate with a reference. It greedily finds a semantic match for each token, so related terms such as “attorney” and “lawyer” can contribute even without an exact word match. BERT is a model; BERTScore is an evaluation algorithm that uses encoder models. section 3.2.2 section 3.2.5 section 3.5
Semantic similarity is not a complete quality assessment. BERTScore requires a reference and can miss the distinction between a minor stylistic difference and a critical factual error. Semantic metrics also cost more to compute, are harder to interpret, and can inherit language-model biases. section 3.2.5 section 2.3.6






