Validation set
A validation set is data set apart from model-fitting data to assess model behavior or guide choices during development.
Timeless Algorithms, The Foundational Ideas
Held-out data used to assess model performance and guide model selection during development.
- How it is used
- Guides architecture, hyperparameter, and early-stopping decisions using observations excluded from parameter fitting.
- How deep it goes
- Distinguishes training, validation, and test roles, then details fold-based SVM evaluation without guaranteeing an optimal model.
- Examples
- Compares neural-network configurations and candidate SVM penalty values using held-out observations or folds.
- What you can do after
- Apply it to compare candidate settings and reserve an independent test set for final evaluation.
Read how Timeless Algorithms, The Foundational Ideas explains validation set →
Building LLM Applications with DSPy
A validation set is a dataset partition used to tune choices or configurations during model development.
- How it is used
- Uses validation examples to select candidate prompts during language-model application optimization.
- How deep it goes
- Explains dataset roles, split proportions, sizing guidance, and automatic splitting without detailing optimizer implementation.
- Examples
- Illustrates automatic splitting with MIPRO and contrasts language-model dataset proportions with a traditional machine-learning split.
- What you can do after
- Apply it via DSPy to select prompts while reserving separate data for development and final testing.
Read how Building LLM Applications with DSPy explains validation set →
Building Reliable AI Systems
A held-out subset of data used to assess a model during development without fitting its parameters on that subset.
- How it is used
- Monitors held-out performance during LLM fine-tuning to inform training decisions.
- How deep it goes
- Describes split sizing, representativeness, leakage checks, and epoch-level metrics without specifying a splitting algorithm.
- Examples
- Uses Q&A fine-tuning to illustrate the risk of memorizing specific answers.
- What you can do after
- Operate training by monitoring validation metrics and considering when to stop.
Read how Building Reliable AI Systems explains validation set →
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
A dataset used to assess a model during development, often to guide model selection or tuning.
- How it is used
- Uses validation data to tune models and assess sampling or labeling during active-learning iterations.
- How deep it goes
- Distinguishes four validation distributions and explains activation ranking with interpolation; leaves the fourth set's purpose unspecified.
- Examples
- Illustrates an iteration baseline against random sampling and interpolates an unlabeled activation between ten validation values.
- What you can do after
- Evaluate model tuning and sampling with validation data matched to the question being asked.
AI Model Evaluation
A validation set is a dataset reserved for assessing or comparing model behavior during development rather than fitting model parameters.
- How it is used
- Uses development data to tune models and compare iterations, or tests an LLM judge against reference ratings before scaling evaluation.
- How deep it goes
- Defines validation’s role and limits, then details a judge-checking loop with reference labels, agreement measures, disagreement analysis, and version locking.
- Examples
- Uses recommendation explanations that are accurate, hallucinated, irrelevant, or persuasive but ungrounded, alongside historical production data for offline evaluation.
- What you can do after
- Evaluate an LLM judge against human references, diagnose disagreements, revise its setup, and lock a version for scaled use.
Grokking Statistics
A validation dataset is a held-out dataset used to evaluate model choices after training and iterative testing but before final assessment.
- How it is used
- Protects final model evaluation from clues leaked through repeated testing and model changes.
- How deep it goes
- Defines the dataset’s role and rationale but does not detail splitting procedures or implementation.
- Examples
- Uses repeated testing and model changes that accidentally leak clues from the test dataset.
- What you can do after
- Evaluate tested models on data excluded from iterative testing before trusting their reported performance.
Also covered in
- Rearchitecting LLMsIn early access (MEAP)
- Evaluating AI SystemsIn early access (MEAP)
- CUDA for LLMs
- Machines that ThinkIn early access (MEAP)
What to know first
- foundation modelA foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- promptA prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
- activation functionAn activation function transforms a neural-network unit’s input into its output, often introducing nonlinearity.
Related concepts
- GEPAused by · GEPA (Genetic-Pareto) is a reflective evolutionary optimization method that generates and evaluates candidate prompts using task information and feedback.
- hyperparameter tuningused by · Hyperparameter tuning is the systematic selection of configurable model or learning-procedure settings by evaluating alternatives against a chosen objective.
- LLM-as-a-judgeuses · LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
- early stoppingused by · Early stopping is a technique that terminates an iterative process before its planned completion when a specified condition indicates that further work is unnecessary or unlikely to improve the objective.
- AI modeluses · An AI model is a computational system that learns or encodes patterns to map inputs to outputs such as predictions, classifications, or generated content.
- BootstrapFewShotWithRandomSearchused by · BootstrapFewShotWithRandomSearch is a DSPy optimizer that evaluates multiple candidate demonstration sets and selects the one with the best validation performance.
Questions about validation set
What is validation set?
A validation set is data set apart from model-fitting data to assess model behavior or guide choices during development.
Which Manning books explain validation set?
Timeless Algorithms, The Foundational Ideas, Building LLM Applications with DSPy, Building Reliable AI Systems, Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI, AI Model Evaluation and Grokking Statistics.
What should I know before learning validation set?
Manning books that teach validation set usually introduce foundation model, fine-tuning, prompt and activation function first.
What is validation set also called?
Validation set is also called validation data or validation dataset.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





