Semantic similarity is the degree to which texts express related meanings, even when their wording differs. It helps evaluate generated text when a valid answer can paraphrase a reference rather than repeat it. Lexical metrics such as BLEU and ROUGE count overlapping words or sequences, so they can penalize a correct paraphrase. section 2.3.6 section 3.2
Meaning versus word overlap
Against the reference “people like foreign cars,” the candidate “Consumers prefer imported cars” preserves the meaning with different words. “People like visiting places abroad” shares more surface wording but expresses a different idea. section 3.0
Embedding-based metrics represent text as vectors: similar meanings tend to occupy nearby regions of an embedding space. Context matters too. The representation of “run” can differ between “run a company” and “run a marathon.” Cosine similarity compares vector directions and can be used to assess sentence similarity. section 3.2.3 section 3.2.2
Figure 3.5 Semantic metrics measure the similarity between passages in embedding space. Even when sentences use completely different vocabulary, if they convey similar meaning, their embeddings will be close together. “The attorney filed the brief” and “the lawyer submitted the document” occupy nearby regions despite sharing no words.
The figure illustrates how passages with different vocabulary can occupy nearby regions in an embedding space. section 3.2.3
To use semantic similarity in evaluation:
- Choose a reference and decide whether valid outputs may use different wording. For paraphrased summaries or dialogue, semantic comparison can complement word overlap. section 3.2 section 3.4.1
- Choose a comparison method. BERTScore matches words using contextual embeddings; sentence similarity can instead use pooled sentence embeddings and cosine similarity. section 3.2 section 3.2.2
- Inspect important facts separately and interpret the score alongside other evidence. A high similarity score can still hide a changed number or negation. section 3.4.2 section 3.5
Semantic metrics cost more to compute and can be harder to interpret than lexical metrics. They can also inherit model biases or reward flaws in a reference. Where exact structure matters, as in code generation or SQL extraction, lexical checks remain useful. COMET is a different learned approach: it predicts human quality judgments and can use the source, candidate, and reference to assess meaning preservation. section 3.4.2 section 3.4.1







