Evaluation turns “this prompt seems better” into a comparison against defined quality criteria. Start with a baseline, score it, and use the result to judge later changes. Evaluation also helps decide whether an optimized application is reliable enough for production; even strong models make errors. section 1.6 section 4.0
For a DSPy program, the basic procedure is:
- Prepare examples for the stage of work: a development set for early checks, a validation set during optimization, and a test set for final evaluation. section 4.2
- Define and test a metric that scores one prediction against one example. Run the program across the chosen set, then aggregate those scores. section 4.2 section 4.8
- Record more than one measure when making a release decision. Accuracy alone can hide slow or expensive responses; consistency and errors also matter. section 4.2.4 section 4.7 section 4.8
Optimization needs a single numeric metric to rank candidate prompts, but final evaluation can use several metrics to reveal trade-offs. Revisit metrics as reviewed production data exposes concerns they missed. A small sample is useful for a quick check, not a dependable estimate: results from 50 examples varied substantially with the data split. section 4.2.4 section 4.3.1
Airline intent classifier
A direct Python loop calls a DSPy classifier for each development example, applies `validate_answer`, and averages the scores. DSPy’s `Evaluate` can perform the same kind of assessment without hand-writing the sequential loop and can run independent examples in parallel. When checking consistency or timing, configure the LM with `cache=False` so cached responses do not distort the result. section 4.3.1 section 4.7
Table 4.1 Comparison by module type and LM
|
Module
|
LM
|
Trial #
|
Accuracy (%)
|
Duration (seconds)
|
Exceptions
|
|---|---|---|---|---|---|
Predict
|
gpt-4o-mini
|
1
|
92.1
|
82.9
|
0
|
|
2
|
93.1
|
102.1
|
0
|
||
|
3
|
93.1
|
77.1
|
0
|
||
Predict
|
gpt-4.1-nano
|
1
|
85.9
|
83.2
|
0
|
|
2
|
86.2
|
121.9
|
0
|
||
|
3
|
86.8
|
86.2
|
0
|
||
ChainOfThought
|
gpt-4o-mini
|
1
|
90.5
|
167.6
|
0
|
|
2
|
93.8
|
205.4
|
0
|
||
|
3
|
92.8
|
196.5
|
0
|
||
ChainOfThought
|
gpt-4.1-nano
|
1
|
88.9
|
78.5
|
0
|
|
2
|
89.5
|
108.6
|
0
|
||
|
3
|
89.6
|
84.6
|
0
|
The repeated classifier trials compare accuracy, duration, and exceptions across two LMs and two DSPy modules. Here, Predict performs about as well as ChainOfThought with gpt-4o-mini, illustrating why a more elaborate module is not automatically the better choice. section 4.7
Keep comparisons fair by evaluating alternatives against the same examples and metric. A manually written OpenAI prompt reached similar accuracy to an unoptimized DSPy prompt in one small comparison; the score did not establish a meaningful winner. Evaluation is a way to test choices, not a guarantee that any framework-generated prompt will outperform a manual one. section 4.4
