EvaluationResult holds the overall score from a DSPy evaluation and a results list containing one tuple per example: the original example, its prediction, and its score. section 4.3.2
The overall score is the average of the individual example scores, so the result brings the summary metric together with the details behind it. section 4.3.2
Intent classification evaluation
A baseline evaluation reports 45 correct predictions out of 50 examples (90%) and returns an EvaluationResult with the overall score and per-example outcomes. section 4.3.2
Listing 4.13 Evaluating the baseline with DSPy Evaluate
import dspy
import time
import os
os.environ['OPENAI_API_KEY'] = OPENAI_API_KEY #1
def evaluate_baseline(model="openai/gpt-4o-mini", num_threads=10):
lm = dspy.LM(model, cache=False)
dspy.configure(lm=lm)
classifier = dspy.Predict(IntentSignature) #2
evaluator = dspy.Evaluate(devset=dev_set, #3
num_threads=num_threads,
display_progress=True,
display_table=5,
provide_traceback=True,
max_errors=100000)
start_time = time.time()
results = evaluator(classifier, metric=validate_answer) #4
end_time = time.time()
print(f"total time: {end_time - start_time}")
return results.score
overall_score = evaluate_baseline(num_threads=10)
See the DSPy Evaluate setup that produces the result; the example stores the returned value and reads its score. section 4.3.2
