Semantic precision measures whether ideas in a generated summary match those in a reference summary, regardless of wording. Unlike lexical precision, it compares meaning rather than words or n-grams. section 8.6.1 section 8.7
Semantic precision and groundedness assess different things: precision asks whether a summary covers relevant ideas, while groundedness asks whether its ideas appear in the original document. Pairing them can help distinguish relevant content from invented content. section 8.6.1
Listing 8.13 Metric function using SemanticF1
import dspy
from dspy.evaluate import SemanticF1
lm = dspy.LM('openai/gpt-4.1-nano', cache=False)
dspy.configure(lm=lm)
test_example = dspy.Example(
question="What is the Rime of the Ancient Mariner about?",
response="A sailor stops a wedding guest to recount his cursed journey
after he "
"thoughtlessly shoots an albatross, a bird of good omen, bringing
death to "
"shipmates and himself.") #1
test_pred = dspy.Prediction(response=
"A mariner recalls his voyage at sea, which was doomed
after he killed a bird") #2
semantic_f1 = SemanticF1() #3
semantic_f1(test_example, test_pred) #4
The DSPy example compares a reference and prediction about the Rime of the Ancient Mariner with SemanticF1. Its examples require fields named question and response, which can be adapted to other field names. section 8.6.1
