Lexical similarity measures how much a candidate text overlaps with a reference at the token or word-sequence level. It treats words as atomic symbols, so matching n-grams increase similarity while synonyms may not. section 3.1.1
Use lexical similarity when surface form is part of correctness:
- Apply it to translation, code generation, SQL extraction, and constrained text generation, where exact phrasing can matter. section 2.1.8 section 3.1.1
- Use BLEU for matching n-grams, ROUGE for reference-n-gram recall, or edit distance for transformation operations. section 3.1.1
- Do not treat the score as semantic understanding: lexical metrics can miss valid paraphrases and cannot assess meaning or coherence. section 3.1.1 section 2.2.3
Attorney and lawyer
A lexical metric treats “attorney” and “lawyer” as different strings, just as it treats “attorney” and “banana” as different strings. Semantic metrics can recognize the first pair as related through learned representations. section 3.1.1
Figure 3.1 The families of automatic evaluation metrics for open-ended tasks. Lexical metrics count word overlaps. Semantic metrics measure meaning similarity using embeddings. LLM-as-a-judge uses LLM as humans to assess directly.
Compare lexical metrics with semantic metrics and LLM-based judging when selecting an evaluation approach. section 3.1.1
