A metric function scores one language-model prediction against one example. In DSPy, it supplies the criterion used to evaluate a program and can also guide optimization. A simple intent classifier can compare its predicted label with the example’s correct label; open-ended outputs require a more carefully designed score. section 4.2.2 section 8.0
A DSPy metric function has three parameters:
- An Example containing the input and any reference value needed for scoring. section 4.2.2
- A Prediction containing the program’s output fields. section 4.2.2
- An optional trace, defaulting to None. It is None during evaluation; optimization can use traces of language-model calls and responses. section 4.2.2
Listing 4.8 Custom evaluation function
def validate_answer(example: dspy.Example, prediction: dspy.Prediction, trace=None): return example.intent_label == prediction.intent_label
The basic implementation returns whether the example and prediction have identical intent labels. If case or surrounding whitespace should not count as errors, normalize the strings before comparing them instead. section 4.2.2
Test the metric as ordinary Python before trusting an evaluation result. Construct Example and Prediction objects directly, check that a matching label returns True, and check that a different label returns False. For a more complex metric, test a diverse set of examples and several plausible predictions per example; keep those tests so changes to the metric can be checked later. section 4.2.5
A metric also decides which generated demonstrations BootstrapFewShot accepts. If a metric checks only the intent label, a passing prediction may contribute all its output fields—including reasoning the metric never checked. A numeric metric can return a score for evaluation and a Boolean acceptance decision when a trace is present; a metric threshold offers another way to set an acceptance cutoff. section 5.3 section 5.3.1
For summaries, a single label match cannot capture qualities such as accuracy, relevance, completeness, and wording. A metric can call a separately evaluated judge program to score a summary, or measure the error between a judge’s predicted groundedness and a known score. Scores used for optimization should follow DSPy’s higher-is-better convention. Multiple metrics can give a fuller evaluation than one score alone. section 8.0 section 8.6.10 section 8.6.6 section 4.8
