Validation set
A validation set is held-out data used to assess a model during development and guide model selection or tuning, rather than to fit its parameters.
Building LLM Applications with DSPy
A validation set is a dataset partition used to tune choices or configurations during model development.
- How it is used
- Supplies DSPy optimizers with data for choosing among generated prompts.
- How deep it goes
- Defines its optimization role and gives split sizes and automatic-splitting behavior, without implementing an optimizer.
- Examples
- Contrasts a zero-shot ATIS intent-classification baseline using test data with later validation-dependent optimization.
- What you can do after
- Apply it via DSPy optimizers to select prompts while keeping development and test data for separate performance checks.
Read how Building LLM Applications with DSPy explains validation set →
Building Reliable AI Systems
A held-out subset of data used to assess a model during development without fitting its parameters on that subset.
- How it is used
- Checks fine-tuning performance on held-out examples during training.
- How deep it goes
- Specifies split size, distribution, metrics, stopping signals, and leakage precautions without deriving the metrics.
- Examples
- Uses a 10–20% split and rising validation loss as concrete planning and monitoring examples.
- What you can do after
- Apply it via a validation file in OpenAI’s fine-tuning API or metrics computed in custom training code.
Read how Building Reliable AI Systems explains validation set →
Grokking Statistics
A validation dataset is a held-out dataset used to evaluate model choices after training and iterative testing but before final assessment.
- How it is used
- Protects final model evaluation from clues leaked through repeated testing and model changes.
- How deep it goes
- Defines the dataset’s role and rationale but does not detail splitting procedures or implementation.
- Examples
- Uses repeated testing and model changes that accidentally leak clues from the test dataset.
- What you can do after
- Evaluate tested models on data excluded from iterative testing before trusting their reported performance.
Related concepts
- GEPAused by · GEPA (Genetic-Pareto) is a reflective evolutionary optimization method that generates and evaluates candidate prompts using task information and feedback.
- evaluatoruses · An evaluator is a model or component that assesses outputs against specified criteria and produces judgments, scores, comparisons, or feedback.
- early stoppingused by · Early stopping is a technique that terminates an iterative process before its planned completion when a specified condition indicates that further work is unnecessary or unlikely to improve the objective.
- cross-validationused by · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
- development datasetcontrasts with · A development dataset is a dataset partition used to iteratively develop, inspect, and compare system configurations before final testing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.