A metric scores the quality of a language-model response. For an application that classifies airline messages, it turns each predicted intent into a result that can be compared across examples. Those results help compare prompts and estimate whether the classifier is reliable enough for production. section 4.0 section 4.2.1
Choose a score that reflects what matters for the task:
- For straightforward classification, return whether the predicted label matches the expected label. section 4.2.1
- For meaningful near misses, return a numeric score: similar intents might earn partial credit, while unrelated intents earn none. Higher scores mean better responses. section 4.2.1
- If spelling or capitalization differences should not count as full errors, account for them when scoring. section 4.2.1
In DSPy, a custom metric is a Python function that receives an Example and the LM response for that Example, then returns a score. Defining the rule is often harder than writing the function: it determines which responses count as good. section 4.2.1
Scoring an intent classifier
If the expected intent is “Flight Booking Request,” exact-match scoring accepts that label and rejects every other one. A graded metric could instead give partial credit when “Airport Location Inquiry” is confused with “Airport Distance Inquiry,” but no credit for confusing unrelated intents. section 4.2.1
Automated prompt optimization needs one numeric metric to rank candidates. Final evaluation should often check several dimensions separately: a prompt that improves accuracy might also be slower or more expensive. Revisit metrics as reviewed production data reveals concerns the original score missed. section 4.2.4
