A validation set is the data an optimizer uses to choose among candidate prompts. In a DSPy application, the training set supplies possible prompt demonstrations, while the validation set helps determine which generated prompt works best. section 4.1.2
Keep tuning and performance checks distinct when enough data is available:
- Use a development set to check whether changes to language models, modules, or optimizers improve the application without repeatedly judging them on validation data. section 4.1.2
- Reserve a hold-out test set for a final estimate of performance, ideally using it only once. section 4.1.2
DSPy represents each split as a Python list of Example objects. A practical starting range is about 30–300 examples each for training and validation, with more of them reserved for validation. An explicit split is not always required: many optimizers split supplied training data automatically; MIPRO, for example, uses a 20/80 training-to-validation split. section 4.1.2 section 4.1.3
When validation data is needed
A zero-shot ATIS intent-classification baseline uses no prompt demonstrations or optimization, so it can begin with test data alone. Once prompt optimization begins, a validation set is needed to compare the optimizer’s choices. section 3.2
