liveBook · Manning

Semantic similarity

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Semantic similarity is the degree to which texts express related meanings despite possible differences in wording.

Semantic similarity is the degree to which texts express related meanings, even when their wording differs. It helps evaluate generated text when a valid answer can paraphrase a reference rather than repeat it. Lexical metrics such as BLEU and ROUGE count overlapping words or sequences, so they can penalize a correct paraphrase. section 2.3.6 section 3.2

Meaning versus word overlap

Against the reference “people like foreign cars,” the candidate “Consumers prefer imported cars” preserves the meaning with different words. “People like visiting places abroad” shares more surface wording but expresses a different idea. section 3.0

Embedding-based metrics represent text as vectors: similar meanings tend to occupy nearby regions of an embedding space. Context matters too. The representation of “run” can differ between “run a company” and “run a marathon.” Cosine similarity compares vector directions and can be used to assess sentence similarity. section 3.2.3 section 3.2.2

Figure 3.5 Semantic metrics measure the similarity between passages in embedding space. Even when sentences use completely different vocabulary, if they convey similar meaning, their embeddings will be close together. “The attorney filed the brief” and “the lawyer submitted the document” occupy nearby regions despite sharing no words.

The figure illustrates how passages with different vocabulary can occupy nearby regions in an embedding space. section 3.2.3

To use semantic similarity in evaluation:

  • Choose a reference and decide whether valid outputs may use different wording. For paraphrased summaries or dialogue, semantic comparison can complement word overlap. section 3.2 section 3.4.1
  • Choose a comparison method. BERTScore matches words using contextual embeddings; sentence similarity can instead use pooled sentence embeddings and cosine similarity. section 3.2 section 3.2.2
  • Inspect important facts separately and interpret the score alongside other evidence. A high similarity score can still hide a changed number or negation. section 3.4.2 section 3.5

Semantic metrics cost more to compute and can be harder to interpret than lexical metrics. They can also inherit model biases or reward flaws in a reference. Where exact structure matters, as in code generation or SQL extraction, lexical checks remain useful. COMET is a different learned approach: it predicts human quality judgments and can use the source, candidate, and reference to assess meaning preservation. section 3.4.2 section 3.4.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains contextual embeddings, cosine similarity, word matching, and metric selection, including factual and interpretability limitations.
Examples
Compares foreign-car paraphrases and contrasting sentences, and illustrates contextual meaning with running a company versus running a marathon.
What you can do after
Evaluate or understand generated text by selecting semantic and lexical metrics for the task and inspecting consequential factual errors separately.
How it connects
Connects semantic similarity to representation learning, contextual embeddings, cosine similarity, lexical matching, and human evaluation.

Also explained in

Compare how 8 books explain semantic similarity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.