Trying prompts by hand makes it hard to tell whether an apparent improvement will hold across user messages—or even across repeated responses to the same message. An evaluation dataset supplies a consistent set of inputs for testing a language-model system. Pair it with a metric that defines a good response, and prompt comparisons become repeatable rather than anecdotal. section 1.2.5 section 4.0
Build the dataset around the application, not just the examples that are easiest to collect:
- Include inputs resembling production traffic and cover the classes the system will encounter. For an intent classifier, those classes are user intents. section 4.1
- Combine available real inputs with examples written by people or generated by an LM. Each source can miss cases or introduce its own limitations. section 4.1
- Add labels when the metric needs them. Classification needs known intents; labels may require manual work or LM labeling checked by a person. Other DSPy evaluations do not always need labeled data. section 4.1
- Choose size deliberately. Tens or hundreds of examples are often enough to start, while larger sets can improve estimates and tuning at the cost of collection effort, runtime, and expense. section 4.1
Keep the dataset’s purpose clear. A test set can provide a final performance estimate; prompt optimization also calls for a separate validation set. A zero-shot baseline needs no training demonstrations or validation set before optimization begins. Where outputs vary between calls, testing the same input more than once can reveal inconsistency, though doing so increases the evaluation workload. section 3.2 section 4.0
Airline intent classification
An airline chatbot’s evaluation inputs are user messages, with intent labels such as flight or airfare providing the information needed to assess classification. A prepared ATIS test-data frame has `text` and `intent` columns. Checking class frequencies exposes the dataset’s mix: flight accounts for 632 of 893 rows before subsequent filtering. Inspecting a few rows helps catch content that does not look right. section 4.1 section 3.2
Listing 3.1 Downloading the dataset
from datasets import load_dataset
import pandas as pd
ds = load_dataset("tuetschek/atis")
ds.set_format(type='pandas')
df = ds['test'][:]
The listing loads the ATIS test split into a pandas data frame, providing a concrete starting point for inspecting evaluation inputs and labels. section 3.2
