A validation set supports development decisions without fitting model parameters. Use it to tune a model, compare iterations, select features or hyperparameters, and check for overfitting. Because repeated use can shape decisions, it is not an independent final assessment; keep holdout data separate for that purpose. Offline results also depend on whether the data represents real user behavior. section 1.3.3
A validation set can also test an evaluation system rather than a predictive model. For an LLM judge, compare its ratings on the same examples against human ratings or gold labels. Inspect disagreements, refine the rubric, prompt, context, or judge model, then lock the version and rerun a fixed canary set when anything changes. section 9.8
Validating a recommendation-explanation judge
Include accurate, hallucinated, irrelevant, and persuasive-but-ungrounded explanations. Compare the judge’s ratings with human references, then inspect disagreements for missed factual errors, rewarded verbosity, missing context, or an ambiguous rubric. section 9.8





