liveBook · Manning

Word-matching metric

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A lexical metric evaluates generated text by measuring overlap or other relationships among surface-level tokens, words, or n-grams.

A word-matching metric evaluates generated text by comparing its surface form with a reference. It is useful when exact wording matters, but a low overlap score can also penalize a correct paraphrase. section 3.1.1 section 3.0

Different lexical metrics measure different relationships:

  • BLEU counts matching n-grams, or sequences of words. section 3.1.1
  • ROUGE measures recall of reference n-grams. section 3.1.1
  • Edit-distance metrics count the operations needed to transform one text into another. section 3.1.1

A correct paraphrase

Against the reference “people like foreign cars,” the candidate “consumers prefer imported cars” conveys the same meaning with different words. BLEU and ROUGE penalize that vocabulary mismatch, so their scores alone do not establish whether the response is correct. section 3.0

Reordering and abstraction can cause similar failures. A sentence can preserve the same information in a different order, breaking n-gram matches. A summary can also express an important insight without repeating the reference’s wording or numbers. section 3.1

Use lexical metrics when exact phrasing or structure matters, as in code generation or SQL extraction, and when fast, interpretable scoring is valuable. Inspect the matches rather than treating a score as a universal measure of quality. For open-ended summaries or dialogue, consider semantic metrics such as BERTScore alongside lexical scores. section 3.1.1 section 3.4.1

Strict matching can also provide a reward for training when a task has a definitive string or numeric answer. BLEU has been used as a reward against a high-quality reference for instruction following, where overlap acts as a proxy for facts and formatting rather than a direct measure of meaning. section 2.3.5

Figure 3.1 The families of automatic evaluation metrics for open-ended tasks. Lexical metrics count word overlaps. Semantic metrics measure meaning similarity using embeddings. LLM-as-a-judge uses LLM as humans to assess directly.

Compare lexical matching with semantic similarity and LLM-as-a-judge evaluation. section 3.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines overlap and edit-distance approaches and diagnoses semantic failures, without detailing score formulas or implementations.
Examples
Contrasts equivalent car descriptions and uses reordered committee statements and financial summaries to expose matching failures.
What you can do after
Evaluate when lexical scores suit exact-output tasks and when paraphrasing calls for semantic metrics.
How it connects
Connects word matching to BLEU, ROUGE, edit distance, semantic similarity, BERTScore, and reinforcement learning with verifiable rewards.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.