Chapter 4: Evaluating DSPy programs

Introduction

The introduction explains why baseline evaluation matters and what makes comparisons of LM prompts reliable.

• Two purposes of evaluation: Explain how evaluation helps select an LM and prompt during optimization and estimate whether an application is reliable enough to release.
  Why it matters: A score is useful both for choosing what to improve and for deciding how much to trust the result.
• Test data and meaningful metrics: Describe why comparing prompts requires a substantial test set and a metric that expresses what counts as a better response.
  Why it matters: Without enough representative tests and a definition of quality, choosing a prompt can come down to an unreliable impression.
• Output variability and repeated testing: Explain why some LM settings call for repeated tests of the same prompt and message, and why automation helps.
  Why it matters: One response may not represent how a variable LM will behave on the next call.

Key terms: evaluation, optimization, baseline model, test data, evaluation metric, test set, temperature, consistency, randomness

4.1 Creating a dataset for evaluation

This section explains how to assemble useful evaluation data for an LM application, including coverage, size, labeling, data sources, and risks to reliable estimates.

• Production-relevant test coverage: Explain why evaluation data should resemble production data or at least include the classes the application will encounter.
  Why it matters: A score on unrepresentative examples may say little about how the application will perform for real users.
• Test set size tradeoff: Describe the benefit and cost of a larger test set, and state the rough amount of data this section says is normally needed.
  Why it matters: The right test-set size balances confidence in the evaluation against the cost of obtaining and using examples.
• Classification data labeling burden: Explain why obtaining usable evaluation examples can be difficult for the airline intent classifier, even if user messages are plentiful.
  Why it matters: A collection of user messages is not automatically an evaluation set for a classifier.
• Complementary evaluation data sources: Justify combining real data, human-created synthetic data, and LM-created synthetic data, including what to do at the start of a project without real data.
  Why it matters: Mixing sources can expose cases that any one source alone would overlook.
• Public dataset evaluation risk: Explain why evaluating on a publicly available dataset can produce an overly optimistic estimate of an LM’s production performance.
  Why it matters: A strong score on familiar public examples may overstate real-world ability.
• Production-driven test set updates: Describe how production examples can improve an existing test set, especially unusual cases and current failures.
  Why it matters: Updating the test set helps it keep reflecting problems encountered in actual use.

Key terms: test data, distribution, classes, user intents, test set, tens or hundreds of examples, labeled data, unlabeled data, airline intents classifier, real data

4.1.1 Using the Example class

This section shows how to represent evaluation cases as DSPy Example objects and assemble them into a test set for the intent classifier.

• Required evaluation example fields: Explain why each Example needs the signature’s input fields and the output fields used for scoring. For the intent classifier, identify message and labels as inputs and intent_label as the true label used to evaluate the response.
  Why it matters: The evaluator needs both the information to run a case and the answer against which to check it.
• Example input field designation: Describe what with_inputs() indicates on an Example and how that relates to the fields sent in LM calls.
  Why it matters: Marking input fields separates what the model receives from information retained to evaluate its answer.
• Test set Example construction: Describe how to build a test set as a list of Example objects, including conversion from simpler tuples or a pandas dataframe. Explain why the classifier’s possible labels appear in each Example even when they are constant across cases.
  Why it matters: This turns existing test data into cases the DSPy classifier can actually run and evaluate.

Key terms: Example, signature, message, labels, intent_label, with_inputs(), input fields, test set, list of tuples, list comprehension

4.1.2 Dividing the data into train and test sets

The section explains how to divide data for DSPy optimization and evaluation while keeping performance estimates reliable.

• Training and validation roles: Explain that DSPy uses training examples as demonstrations in prompts and validation examples to choose among generated prompts. Distinguish this from traditional ML, where training data creates the model, while validation still guides tuning.
  Why it matters: Keeping these roles distinct makes it clearer what the optimizer is using and what it is measuring.
• Separate development set: Explain why this example uses a development set in addition to a validation set: validation helps optimizers tune prompts, while development tracks whether the baseline and later versions improve without repeatedly judging them on the tuning data.
  Why it matters: A separate development check makes apparent progress less likely to be just overfitting to validation data.
• Hold-out test set: Justify saving the test set for a final, ideally one-time evaluation instead of using it to assess every iteration.
  Why it matters: The final test is most useful when earlier decisions have not been shaped by its results.
• Representative evaluation data: Describe how new production cases might be assigned to the sets, while keeping development and test data broadly representative of real-world cases.
  Why it matters: Evaluation sets that resemble production give a more realistic picture of application quality.
• Cross-validation trade-off: Explain how cross-validation repeatedly creates different training and validation splits, why that can produce more reliable estimates than one split, and why LM-call costs make it less common for LM applications.
  Why it matters: This explains why a convenient single split can give a fragile result, even when it is the practical choice.

Key terms: training set, validation set, demonstrations, optimizer, development set, overfit, test set, hold-out test set, unbiased estimate, real-world cases

4.1.3 The sizes of the sets

This section explains why DSPy can use a modest training pool to choose a few prompt examples and why LM applications may allocate a larger share of their data to validation and testing.

• Training pool and prompt examples: Explain why a DSPy training set is usually larger than the number of few-shot examples placed in a prompt, and contrast this with many-shot prompting.
  Why it matters: The number of examples available for selection need not equal the number sent in every prompt.
• LM data split proportions: Describe how training, validation, and test proportions for LM applications can differ from traditional machine learning, including the section’s approximate size recommendation.
  Why it matters: A large validation share can be appropriate even when the dataset contains only dozens or hundreds of examples.
• Automatic optimizer validation split: Explain what most DSPy optimizers do when given a training set without a separate validation set.
  Why it matters: This clarifies that omitting an explicit validation set does not necessarily mean optimization proceeds without one.

Key terms: training set, few-shot examples, many-shot, demonstrations, training, validation, test sets, optimizers, validation set

4.1.4 Splitting the ATIS data

This section shows how the ATIS data is divided into four small sets for evaluation while keeping time and cost down.

• ATIS dataset split: Describe which Hugging Face ATIS split supplies each of the four sets, how many rows each contains, and why the example uses only 200 rows.
  Why it matters: The split tells you where each evaluation set came from and why this example uses less data than usual.

Key terms: training set, validation set, development set, test set, ATIS

4.1.5 Using DSPy to generate examples

The section explains two ways to generate synthetic intent-classification examples with DSPy and why the generated data needs checking.

• Label-based example generation: Explain how an LM can generate realistic utterances from intent labels when real-world data is scarce or may not cover production cases. Describe the example of generating five questions for each possible intent.
  Why it matters: Label-based generation offers a way to add examples for intents that may be underrepresented in the available data.
• Synthetic example quality checks: Justify checking generated examples before using them and explain why generating five questions in one LM call is preferred here. Describe the suggested check for duplicates.
  Why it matters: Reviewing and deduplicating generated questions helps keep unsuitable examples out of classifier work.
• Variations of labeled examples: Describe how an LM can create variations of existing, confirmed well-labeled examples while retaining each original example's intent label. State what the example does for each test-set item.
  Why it matters: Variations provide more examples based on cases whose labels have already been confirmed.

Key terms: synthetic data, intents, Predict module, synthetic questions, distinct synthetic utterances, duplicates, variations, test set, intent_label

4.2 Evaluating a module with a test set

This section introduces evaluating a DSPy program by scoring individual predictions and then assigning an overall score for a dataset.

• Individual and overall evaluation: Explain the two levels of evaluation for the airline intent classifier: assess each LM intent prediction against its Example, then assign the program an overall score across the evaluated Examples. State which dataset is used at the current, optimization, and final-evaluation stages.
  Why it matters: It keeps the quality of a single prediction distinct from the program’s score on a dataset.

Key terms: Examples, development set, validation set, test set, overall score

4.2.1 Defining an evaluation metric

The section explains how to score individual DSPy responses, using exact-match intent classification as the baseline while showing optional ways to make scoring more nuanced.

• Per-response metric function: Explain that a custom DSPy evaluation metric is typically a Python function that takes an Example and the LM’s response to it, then returns a score for that response.
  Why it matters: This is the basic scoring interface used to evaluate the classifier’s responses.
• Exact-match classification scoring: Describe why this intent classifier can use a Boolean metric: each utterance has a ground-truth class, so the prediction is scored True only when it exactly matches that class. Distinguish this from tasks such as summarization or style transfer, which may lack a single ground-truth response.
  Why it matters: A clear correct label lets you start with a simple, testable classifier metric.
• Nuanced numeric scoring: Explain how a metric could award partial numeric credit for confusing similar intents or for a substantially correct label with minor spelling or capitalization differences, while giving less credit to unrelated errors.
  Why it matters: Partial credit can reflect meaningful differences between errors that exact matching treats alike.
• Confidence-aware prediction scores: Describe how predicted confidence could affect a classification score: reward high confidence more when the label is correct and penalize it more when the label is wrong. Explain that an ideal confidence may also be included in an Example when there is ground truth for it.
  Why it matters: Confidence-aware scoring distinguishes a cautious mistake from a highly confident one.

Key terms: evaluation metric, Example, LM’s response, score, classification, Boolean, ground truth, True or False, floating-point number, fuzzy matching

4.2.2 Defining a metric for the baseline model

This section defines the inputs to a DSPy evaluation metric and explains why the airline chatbot baseline uses exact-match scoring.

• Metric function inputs: Describe the three parameters a DSPy metric function must accept and what each represents during evaluation.
  Why it matters: These inputs let the metric compare a model response with the corresponding test example.
• Baseline exact-match scoring: Explain how the airline chatbot’s baseline metric checks an answer and why the section uses exact matching rather than the more flexible alternative.
  Why it matters: The matching rule determines which chatbot responses count as correct in the baseline evaluation.

Key terms: metric function, Example, Prediction, trace, signature, gold values, exact match, validate_answer()

4.2.3 Calculating a final evaluation for the module

The section explains how to turn per-example test scores into an overall evaluation that reflects what matters for the application.

• Numeric score aggregation: Describe how to calculate an overall score from numeric per-example scores, and why the mean is a simple starting point rather than the only choice.
  Why it matters: An overall number is useful only if the way it combines scores suits the task.
• Average versus weak responses: Explain why the higher-average program in the example is not automatically preferable, and how an overall score can emphasize avoiding poor responses.
  Why it matters: The score used to compare programs should reflect whether strong average results or avoiding weak responses matters more.
• Boolean score accuracy: Explain what averaging Boolean correctness scores measures in the airline chat example, and why that overall figure can be misleading.
  Why it matters: A simple accuracy figure is easy to interpret, but it may obscure performance when some classes occur much more often than others.

Key terms: per-example scores, mean average, median score, minimum score, low percentile, harmonic average, outlier, Boolean scores, accuracy, class imbalance

4.2.4 Evaluation for tuning versus final evaluation

The section explains why automated prompt tuning uses one numeric metric, while final evaluation should examine several dimensions and metrics should improve as new problems emerge.

• Tuning and final evaluation metrics: Distinguish the single numeric metric needed to compare candidate prompts during automated optimization from the multiple metrics that may be used to assess a final program before production.
  Why it matters: A strong tuning score alone may not tell you whether the finished application is suitable for production.
• Single metric blind spots: Explain how optimizing for one metric can conceal or worsen other qualities of an LM application, using the section’s accuracy versus speed and cost example.
  Why it matters: Checking for trade-offs helps prevent an impressive score from hiding a poor user or production outcome.
• Iterative metric improvement: Describe how reviewed production data can reveal undesirable responses that current metrics miss, prompting updates to those metrics for later optimization.
  Why it matters: Revisiting metrics lets future optimization address problems that the original evaluation overlooked.

Key terms: metric, optimization, final evaluation, test set, Goodhart’s law, accuracy, concise, cost effective, production, metrics

4.2.5 Testing the metric function

This section shows how to unit-test an evaluation metric directly and how to choose test cases that give confidence in its scores.

• Direct metric function tests: Describe how to test a metric function by passing it an Example and a Prediction and checking the returned score, including both matching and nonmatching intent labels.
  Why it matters: Checking the metric itself helps ensure later scores reflect the behavior you intend to measure.
• Representative metric test cases: Explain why metric tests should be maintained and cover diverse example–prediction pairs, especially when a task permits many plausible responses.
  Why it matters: A varied, reusable test set makes incorrect metric scores less likely to undermine optimization or final evaluation.

Key terms: evaluation metric, Example, Prediction, intent_labels, unit tests, metric functions, weak, moderate, and strong responses

4.3 Evaluating the DSPy baseline model

This section identifies the unoptimized DSPy baseline program to be evaluated and notes that evaluation can be done manually or through DSPy.

• Unoptimized baseline program components: Explain why the baseline program being evaluated consists of an LM, signature, and module, without optimization.
  Why it matters: It clarifies exactly what is being evaluated before optimization changes the program.

Key terms: LM, signature, module, optimization

4.3.1 Using custom Python code for evaluation

This subsection shows how to evaluate a baseline classifier with a manual Python loop and explains the practical limits of that approach.

• Manual baseline evaluation loop: Trace how the code evaluates the baseline: run the classifier on each development-set example, score its prediction with validate_answer(), and average the per-example scores.
  Why it matters: This makes clear what the evaluation score represents before using an evaluation framework.
• Example input field unpacking: Explain that **example.inputs() passes the fields selected by with_inputs() as classifier parameters; here those fields are message and labels.
  Why it matters: It helps the reader follow how each stored example is passed to the classifier.
• Uncached evaluation responses: Explain why the example creates the LM with cache=False: it prevents reuse of responses to previously seen queries, which could distort consistency evaluations and execution-time measurements.
  Why it matters: Cached responses can make an evaluation measure something other than fresh model behavior or execution time.
• Evaluation sample size tradeoff: Describe why the example uses only 50 development-set examples and why that score should not be treated as a stable estimate of program skill.
  Why it matters: It helps distinguish a quick demonstration score from a reliable estimate of performance.
• Sequential evaluation limitation: Explain why this working manual approach is inefficient: it tests independent examples sequentially, although they could be evaluated in parallel; adding that parallelism by hand would duplicate work DSPy provides.
  Why it matters: It explains the motivation for replacing the hand-written evaluation loop.

Key terms: dev_set, Predict, validate_answer(), per-example scores, Example, **example.inputs(), with_inputs(), message, labels, cache=False

4.3.2 Using DSPy Evaluate

This section shows how DSPy Evaluate runs and reports the baseline evaluation, and explains its advantages and trade-offs compared with a manual loop.

• Evaluate setup and automatic scoring: Describe how to create and run an Evaluate evaluator with a dataset, thread count, and display settings. Explain that Evaluate calls the DSPy program for each example and returns the average of the individual evaluation scores.
  Why it matters: You can evaluate a program without writing the loop or calculating its overall score yourself.
• Framework consistency and parallelism: Explain why the section favors DSPy’s evaluation framework over a hand-written loop for routine evaluation, including cleaner, more consistent evaluation and parallel LM calls.
  Why it matters: A standard, multithreaded evaluator can make repeated LM evaluations easier to run and compare.
• Evaluation results and display: Describe what the returned EvaluationResult contains and how the optional displayed table relates to its per-example results.
  Why it matters: The returned results let you inspect individual predictions as well as the overall score.
• Error limits and score bias: Explain what max_errors controls and why a proper evaluation should halt rather than rely on results from too few successful examples.
  Why it matters: An apparently valid score can be misleading if errors prevented many examples from being evaluated.
• Manual evaluation early stopping: Explain the advantage manual evaluation retains when a prompt is already performing poorly.
  Why it matters: When a prompt is clearly unsuitable, a manual loop can avoid spending LM calls on an exact score you do not need.

Key terms: Evaluate, devset, environment variable, final score, evaluation framework, multithreaded, parallelization, EvaluationResult, results, display_table

4.3.3 Rate limits

Explains how provider rate limits can interrupt parallel evaluation and how to adjust evaluation concurrency.

• Rate limits during evaluation: Explain why parallel LM evaluation can hit provider rate limits and halt, and describe how to adjust concurrency while retaining useful parallelism.
  Why it matters: Managing request volume keeps evaluations running without giving up more parallelism than necessary.

Key terms: rate limits, threads, max_errors, sleep statements

4.4 Evaluating a manually created prompt using the OpenAI API

The section compares a manually written OpenAI prompt with a DSPy-generated prompt and interprets their similar evaluation scores.

• Manual prompt comparison: Explain why a manually created prompt can be evaluated alongside a DSPy-generated prompt and how this comparison is kept consistent: the example uses a hardcoded prompt through the OpenAI API with the same LM and test set.
  Why it matters: A comparable baseline helps you judge whether DSPy improves on a prompt you could write yourself.
• Accuracy and effort tradeoff: Describe what the reported accuracy comparison does and does not show, and explain the difference in effort between the manual and DSPy prompts.
  Why it matters: A similar score does not mean the two approaches took the same effort or that one has proved superior.

Key terms: manually created prompt, baseline, hardcoded prompt, test set, OpenAI’s API, accuracy, unoptimized prompts, handcraft the prompt

4.5 Evaluating per-class performance

This section shows how to inspect classification performance by class and calculate a macro F1 score alongside overall accuracy.

• Per-class performance analysis: Explain why a classification report is useful alongside an overall score, and describe how the section uses the test set to find weak classes.
  Why it matters: A good overall score can coexist with poor results for particular classes.
• Macro F1 metric choice: Describe how the section calculates an overall macro F1 score and explain how it differs from average accuracy.
  Why it matters: The overall metric should reflect the kinds of classification performance that matter to the project.

Key terms: per-class performance, classification_report, test records, underperforming classes, macro F1 score, precision, recall, ground truths, predictions

4.6 Evaluating the consistency of responses

The section compares repeated-test methods for judging both a DSPy program’s accuracy and the consistency of its responses.

• Repeated examples and overall accuracy: Explain how duplicating each test example produces an overall accuracy score based on repeated responses, why that can be more informative than one run per example, and what it costs.
  Why it matters: Repeated examples can make an accuracy estimate more meaningful without being mistaken for a direct consistency measure.
• Per-example prediction consistency: Describe how to assess consistency by repeating an example, counting its different predictions, and comparing them with the most frequent prediction.
  Why it matters: A direct consistency count reveals changing answers that an overall accuracy score can hide.
• Conservative scores across repeated tests: Distinguish repeating the entire test set and combining its overall scores from taking the minimum score across repeated runs of each individual example.
  Why it matters: Per-example minimum scores can expose unreliable answers even when repeated whole-test scores look stable.

Key terms: test example, overall accuracy, consistency, predictions, most popular prediction, Evaluate, full test set, minimum score, Boolean scores

4.7 Evaluating other LMs and modules

This section compares LMs and DSPy modules against an evaluated baseline, showing how to weigh accuracy, runtime, exceptions, and randomness when choosing a current-best program.

• Configurable LM and module comparisons: Describe how to reuse evaluate_baseline() to compare different LMs and module types, including Predict and ChainOfThought.
  Why it matters: A configurable evaluator makes it practical to compare candidate programs on the same basis.
• Repeated runs and reliable comparisons: Explain why LM-and-module comparisons should use a substantial test set and repeated runs, and how to interpret close scores.
  Why it matters: It helps prevent a chance result from being mistaken for a real improvement.
• Accuracy and efficiency tradeoffs: Explain what the tested results imply about choosing Predict versus ChainOfThought and retaining the baseline.
  Why it matters: The best choice depends on whether a more elaborate module actually improves the measured result enough to justify its cost.
• Exceptions and rate-limit effects: Describe why exceptions belong in LM comparisons and how rate-limit workarounds affected the reported cross-model results.
  Why it matters: Reliability and testing conditions matter alongside accuracy when interpreting a model comparison.

Key terms: evaluate_baseline(), model parameter, Predict, ChainOfThought, test set, mean, standard deviation, randomness, train-validation-test splits, gpt-4o-mini

4.8 Summary