A gold example is an evaluation case with an expected judgment or score established by human experts, domain experts, or another trusted reference process. It gives an LLM judge a concrete quality anchor instead of leaving scoring behavior entirely to a prompt or model preference. section 9.0 section 9.6.1
Use gold examples for two related but distinct evaluation activities:
- Calibrate the judge by showing what different scores or quality levels look like, such as a score of 5 versus a score of 2. section 9.6.1
- Validate the judge by comparing its outputs with gold labels, human ratings, or known decision criteria; calibration alone does not establish reliability. section 9.11
- Inspect disagreements and monitor consistency across repeated runs before trusting the judge for important decisions. section 9.5.4
Movie recommender calibration
A recommender evaluation can include several movie outputs already scored by humans or domain experts. Supplying these gold-standard cases in the judge prompt anchors its scoring and reasoning to the intended quality definition, reducing arbitrary scoring and subjective drift. section 9.6.1
Gold examples support scalable semantic evaluation, but they do not make an LLM judge objective. Humans still define the rubric, choose representative cases, inspect disagreements, and decide whether the judge is reliable enough for the decision. section 9.0 section 9.3
