Semantic recall

As explained in Building LLM Applications with DSPy

Semantic recall measures how well the ideas in a reference summary are covered by a generated summary, regardless of wording.

import dspy
from dspy.evaluate import SemanticF1

lm = dspy.LM('openai/gpt-4.1-nano', cache=False) 
dspy.configure(lm=lm)

test_example = dspy.Example( 
   question="What is the Rime of the Ancient Mariner about?", 
   response="A sailor stops a wedding guest to recount his cursed journey
             after he "
            "thoughtlessly shoots an albatross, a bird of good omen, bringing
             death to " 
            "shipmates and himself.")  #1

test_pred = dspy.Prediction(response=
   "A mariner recalls his voyage at sea, which was doomed 
    after he killed a bird")  #2

semantic_f1 = SemanticF1()  #3
semantic_f1(test_example, test_pred)  #4

How Building LLM Applications with DSPy covers it

How deep it goes

Distinguishes semantic matching from lexical matching and names SemanticF1, without detailing its calculation.

Examples

Uses a SemanticF1 example comparing two summaries of the Rime of the Ancient Mariner.

What you can do after

Evaluate generated summaries for coverage of ideas rather than overlap in words or n-grams.

How it connects

Connects semantic recall to semantic precision, SemanticF1, lexical precision and recall, and LLM-as-a-judge evaluation.

liveBookmanning.com

Read Building LLM Applications with DSPy on liveBook Get the book at manning.com

Related concepts

• recallkind of · Recall is the fraction of relevant, positive, or target outcomes that a system successfully identifies, retrieves, or captures.
• completenessequivalent to · Completeness is the extent to which all elements required or relevant to a defined scope are present, represented, or accounted for.

This explainer was generated by Manning's knowledge graph from the text of Building LLM Applications with DSPy. The section links point to the parts of the book it draws on.