liveBook · Manning

Embedding

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A vector is an ordered numerical representation of a point, direction, or feature pattern in a mathematical space.

An embedding is a learned representation that maps a discrete object, such as a word, to a continuous vector: an ordered list of numbers locating it in a high-dimensional space. Words used in similar contexts tend to occupy nearby regions, turning some semantic relationships into geometric relationships. section 3.2.1

Meaning beyond word overlap

“The attorney filed the brief” and “The lawyer submitted the document” use different words but convey similar meanings. Embeddings can place related words and sentences near one another, allowing semantic comparison where lexical matching misses the overlap. section 3.2.2 section 3.2.3

One approach to comparing sentences is to look up each word’s vector, average the vectors for each sentence, and calculate cosine similarity between the results. Cosine similarity compares vector directions rather than lengths: identical directions score 1, perpendicular directions 0, and opposite directions −1. section 3.2.2

Static word embeddings assign a word the same vector in every context, so they cannot distinguish a financial bank from a river bank. Averaging also discards word order and syntax. Contextual embeddings instead change with surrounding words: “run” can have different representations in “run a company” and “run a marathon.” section 3.2.2 section 3.2.3

Figure 3.2 Both max pooling and mean pooling are methods to aggregate information across many vectors. Max pooling selects (pools) the maximum values from input embeddings; this weighs towards the most prominent feature. Mean pooling takes the mean (average) of input embeddings to represent features equally.

Mean pooling averages vectors, while max pooling selects the maximum value in each position. These are different ways to aggregate information across embeddings. section 3.2.2

Embeddings support learned evaluation metrics, but the comparison method matters. BERTScore uses contextual embeddings to assess semantic similarity against a reference. COMET creates sentence embeddings for the source, candidate, and reference, then uses their interactions to predict quality. A similarity score should be interpreted in the context of the evaluation rather than treated as a universal measure of quality. section 3.2 section 3.2.3 section 3.3.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains geometric representation, cosine similarity, pooling, contextualization, and COMET’s use of sentence vectors.
Examples
Uses attorney–lawyer paraphrases and the financial versus river meanings of bank to illustrate similarity and context.
What you can do after
Evaluate and understand semantic similarity using sentence vectors while recognizing the limitations of static embeddings and averaging.
How it connects
Connects embeddings to cosine similarity, contextual embeddings, BERTScore, and COMET.

Also explained in

Compare how 72 books explain embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.