liveBook · Manning

Word overlap

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Word overlap is the degree to which two text outputs share the same words or word sequences.

Word overlap measures how many words or word sequences a candidate shares with a high-quality reference. It provides a simple, interpretable proxy for text quality without modeling meaning. section 2.0 section 2.1.1

Evaluation metrics use overlap in several complementary ways:

  • BLEU emphasizes precision: it rewards candidate words and phrases that also appear in references. section 2.0
  • ROUGE emphasizes recall: it rewards reference content recovered by a candidate, making it useful for summarization. section 2.0
  • BLEU combines overlap scores across word-sequence lengths with a geometric mean and length penalty. section 2.0

Translation comparison

For a Chinese news sentence, Candidate 1—“It is a guide to action which ensures that the military always obeys the commands of the party.”—shares phrases such as “It is a guide to action” with the references. Candidate 2 shares fewer recognizable sequences, so overlap helps distinguish the outputs even without understanding Chinese. section 2.1.1

Word overlap does not recognize synonyms or paraphrases: “consumers prefer imported cars” and “people like foreign cars” can express the same meaning while sharing few words. Treat it as a fast sanity check, not a universal quality measure. section 2.0 section 3.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines overlap and traces n-gram counting, precision-recall choices, geometric means, and length penalties, but stops short of semantic matching.
Examples
Uses translation candidates and references, including a Chinese news sentence, to show why shared phrases can distinguish outputs.
What it assumes
Assumes no knowledge of Chinese for the comparison and defines candidates and references before measuring overlap.
What you can do after
Evaluate translation or summarization outputs with interpretable lexical checks while recognizing when paraphrase makes overlap misleading.
How it connects
Connects word overlap to BLEU, ROUGE, lexical metrics, semantic metrics, BERTScore, and COMET.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.