Lexical comparison evaluates generated text against a reference by checking shared words or word sequences (n-grams). For summarization, it is a straightforward way to compare a generated summary with a ground-truth summary. It is relatively easy and inexpensive to set up where word overlap reflects the quality you want to measure. section 8.1.2 section 8.7
When overlap misleads
Suppose a reference says a sailor stops a wedding guest to tell of a cursed voyage. “A mariner recalls his voyage at sea” conveys a similar idea but shares few words. A summary saying the wedding guest stops the sailor reuses more of the reference’s wording while reversing who stops whom. Lexical comparison can favor the less accurate summary. section 8.1.2
Synonyms, pronouns, alternate spellings, and negation can all separate word overlap from meaning. Lexical comparison tends to suit extractive summaries better than abstractive ones, which may express the same ideas in different words. When matching ideas matters more than matching wording, consider embeddings or an LLM-as-a-judge instead. section 8.1.2 section 8.7
Listing 8.1 Examples with multiple plausible summaries
import dspy
validation_set = [
dspy.Example({ #1
"document": "[the text of The Rime of the Ancient Mariner]", #2
"summary_1": "A sailor stops a wedding guest to recount his cursed journey after he thoughtlessly shoots an albatross, a bird of good omen, bringing death to his shipmates and himself",
"summary_2": "A sailor impulsively kills an albatross, leading to a supernatural curse that dooms his ship and crew, leaving him isolated and tormented until he learns to love and respect all of God's creatures."}).with_inputs("document"),
dspy.Example({ #3
"document": "[the text of Howl]",
"summary_1": "Howl passionately decries the destructive forces of society (Moloch) on the best minds of my generation",
"summary_2": "lamenting the destruction of the best minds of his generation by madness, conformity, and societal repression"}).with_inputs("document"),]
Two plausible reference summaries for each document illustrate why overlap with any one reference needs careful interpretation. section 8.1.2
