liveBook · Manning

Lexical similarity

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Lexical similarity is the degree to which texts resemble one another in their surface-level words or sequences of tokens.

Lexical similarity measures how much a candidate text overlaps with a reference at the token or word-sequence level. It treats words as atomic symbols, so matching n-grams increase similarity while synonyms may not. section 3.1.1

Use lexical similarity when surface form is part of correctness:

  • Apply it to translation, code generation, SQL extraction, and constrained text generation, where exact phrasing can matter. section 2.1.8 section 3.1.1
  • Use BLEU for matching n-grams, ROUGE for reference-n-gram recall, or edit distance for transformation operations. section 3.1.1
  • Do not treat the score as semantic understanding: lexical metrics can miss valid paraphrases and cannot assess meaning or coherence. section 3.1.1 section 2.2.3

Attorney and lawyer

A lexical metric treats “attorney” and “lawyer” as different strings, just as it treats “attorney” and “banana” as different strings. Semantic metrics can recognize the first pair as related through learned representations. section 3.1.1

Figure 3.1 The families of automatic evaluation metrics for open-ended tasks. Lexical metrics count word overlaps. Semantic metrics measure meaning similarity using embeddings. LLM-as-a-judge uses LLM as humans to assess directly.

Compare lexical metrics with semantic metrics and LLM-based judging when selecting an evaluation approach. section 3.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines token overlap and contrasts BLEU, ROUGE, and edit distance, but stops short of semantic interpretation.
Examples
Uses n-gram overlap, reference-n-gram recall, edit operations, and the attorney–lawyer synonym contrast.
What you can do after
Apply it via BLEU, ROUGE, or edit distance when lexical form is an appropriate quality proxy.
How it connects
Connects BLEU, ROUGE, edit distance, semantic similarity, representation learning, BERTScore, and COMET.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.