Semantic equivalence means that two texts express the same meaning despite using different words, phrasing, or structure. Lexical overlap cannot reliably detect it because paraphrases and synonyms may share few or no n-grams. section 2.3.6 section 3.0
Distinguish semantic equivalence from lexical matching by checking whether:
- Different expressions preserve the same proposition, as with “the doctor treated the patient” and “the physician cared for the sick person.” section 2.3.6
- A candidate preserves the reference meaning rather than merely repeating its words, as with “consumers prefer imported cars” for “people like foreign cars.” section 3.0
- A paraphrase remains appropriate to its context, such as “I apologize for the inconvenience” versus “Sorry for the trouble.” section 3.0
BERTScore approaches equivalence through contextual embeddings, matching each candidate token with its best semantic counterpart in the reference. IDF weighting emphasizes distinctive content, while baseline rescaling improves score interpretation. COMET instead learns an evaluation function from human judgments and uses source, candidate, and reference inputs to check whether meaning survived translation or generation. section 3.5 section 3.5
A lexical failure
For the reference “people like foreign cars,” “consumers prefer imported cars” is semantically identical, while “people like visiting places abroad” discusses a different subject. BLEU and ROUGE may favor word overlap and penalize the correct paraphrase. section 3.0
|
Design Principle
|
BLEU Implementation
|
Applied
|
|
Start with the simplest proxy
|
N-gram overlap with references
|
Find the easiest measurable property that correlates with quality.
|
|
Anticipate gaming
|
Modified precision (clip counts)
|
Ask “how could a system cheat?” and close loopholes.
|
|
Require balanced performance
|
Geometric mean aggregation
|
Use when failure in any components is unacceptable.
|
|
Penalize degenerate behavior
|
Brevity penalty
|
Add explicit penalties for too-short, too-long, or other inefficiencies and failure modes.
|
Use these metric-design principles to understand why lexical proxies can be useful, how they can be gamed, and where semantic evaluation becomes necessary. section 2.1.8



