A training dataset supplies candidate demonstrations for a language-model prompt. In DSPy, an optimizer can select a few effective examples from a larger training set. Training here means helping the LM follow prompt instructions, rather than requiring every training example to appear in each prompt. section 4.1.2 section 4.1.3
Keep the data roles distinct when optimizing a DSPy application:
- Training examples provide candidate demonstrations; DSPy datasets take the form of Python lists of Example objects. section 4.1.2
- Validation examples help the optimizer choose among generated prompts. A separate development set can check progress without repeatedly relying on validation data. section 4.1.2
- Reserve the test set for a final, preferably one-time estimate of performance. section 4.1.2
The training set can be larger than the prompt: selecting about five few-shot demonstrations might start with 30, 40, or more candidates. Most DSPy optimizers can split a supplied training set when no validation set is given, though separating the sets explicitly makes their roles clearer. Using hundreds of demonstrations is possible but increases prompt cost and can sometimes confuse the LM. section 4.1.3
When training data is not needed yet
An ATIS intent-classification baseline uses a simple zero-shot prompt, with no demonstrations. It can therefore start with the test data alone; training examples become relevant when demonstrations and prompt optimization are added. section 3.2
