A summary can use different words from a reference summary yet express the same meaning. Conversely, nearly identical wording can conceal a changed meaning, especially with negation. Semantic evaluation compares meanings rather than requiring matching words. It is useful when lexical comparison is insufficient, particularly for abstractive summaries. section 8.1.2
Use embeddings or an LLM judge to compare a generated summary with a ground-truth summary. An LLM judge can be prompted to prioritize particular criteria—for example, topics covered and tone, but not writing style. Decide which differences matter before interpreting a similarity judgment. section 8.1.2
Listing 8.1 Examples with multiple plausible summaries
import dspy
validation_set = [
dspy.Example({ #1
"document": "[the text of The Rime of the Ancient Mariner]", #2
"summary_1": "A sailor stops a wedding guest to recount his cursed journey after he thoughtlessly shoots an albatross, a bird of good omen, bringing death to his shipmates and himself",
"summary_2": "A sailor impulsively kills an albatross, leading to a supernatural curse that dooms his ship and crew, leaving him isolated and tormented until he learns to love and respect all of God's creatures."}).with_inputs("document"),
dspy.Example({ #3
"document": "[the text of Howl]",
"summary_1": "Howl passionately decries the destructive forces of society (Moloch) on the best minds of my generation",
"summary_2": "lamenting the destruction of the best minds of his generation by madness, conformity, and societal repression"}).with_inputs("document"),]
The paired summaries illustrate how more than one wording can plausibly describe the same document. section 8.1.2
