liveBook · Manning

Validation set

As explained in AI Model Evaluation

A validation set is a dataset reserved for assessing or comparing model behavior during development rather than fitting model parameters.

A validation set supports development decisions without fitting model parameters. Use it to tune a model, compare iterations, select features or hyperparameters, and check for overfitting. Because repeated use can shape decisions, it is not an independent final assessment; keep holdout data separate for that purpose. Offline results also depend on whether the data represents real user behavior. section 1.3.3

A validation set can also test an evaluation system rather than a predictive model. For an LLM judge, compare its ratings on the same examples against human ratings or gold labels. Inspect disagreements, refine the rubric, prompt, context, or judge model, then lock the version and rerun a fixed canary set when anything changes. section 9.8

Validating a recommendation-explanation judge

Include accurate, hallucinated, irrelevant, and persuasive-but-ungrounded explanations. Compare the judge’s ratings with human references, then inspect disagreements for missed factual errors, rewarded verbosity, missing context, or an ambiguous rubric. section 9.8

How AI Model Evaluation covers it

How deep it goes
Defines validation’s role and limits, then details a judge-checking loop with reference labels, agreement measures, disagreement analysis, and version locking.
Examples
Uses recommendation explanations that are accurate, hallucinated, irrelevant, or persuasive but ungrounded, alongside historical production data for offline evaluation.
What it assumes
Requires historical records for offline model evaluation and human ratings or gold labels to validate an LLM judge.
What you can do after
Evaluate an LLM judge against human references, diagnose disagreements, revise its setup, and lock a version for scaled use.
How it connects
Connects validation data to training data, holdout data, offline evaluation, and overfitting; connects judge validation to human ratings and evaluation rubrics.

Also explained in

Compare how 6 books explain validation set →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.