Validation checks whether an evaluator’s outputs are reliable enough for a specific decision by comparing them with trusted references. For an LLM judge, clean JSON or calibration with gold examples is not proof of reliability. Validation is especially important when scores affect model selection or release decisions. section 9.8 section 9.11
Use a validation loop before scaling an LLM judge:
- Build a small, representative set and collect human ratings or gold labels. Include easy, ambiguous, and failure cases relevant to the task. section 9.8
- Run the judge on those examples and measure agreement with the references. Depending on the task, use accuracy, score correlation, win-rate agreement, or disagreement rates. section 9.8
- Inspect disagreements for missed errors, verbosity bias, missing context, or ambiguous rubric criteria. Revise the rubric, prompt, context, or judge model as needed. section 9.8
- Once performance is sufficient, lock the judge version. Rerun a fixed canary set when anything changes, and monitor judgment behavior over time. section 9.8 section 9.5.4
Validate recommendation explanations
Build a set containing accurate, hallucinated, irrelevant, and persuasive but ungrounded explanations. Compare the judge’s ratings with human ratings or gold labels, then inspect disagreements to find whether the judge rewards persuasion without checking the user’s history. section 9.8
Check stability separately from agreement with references. A judge can be consistently wrong or correct on average but noisy across reruns. If run-to-run variation is wider than the score gap between two model variants, the comparison may reflect noise rather than a quality difference. Repeated sampling exposes general variation; swapped-order reruns can expose position bias. section 10.4.1 section 10.7











































