liveBook · Manning

Gold example

As explained in AI Model Evaluation

A gold example is an evaluation case whose expected judgment or score has been established by human experts or another trusted reference process.

A gold example is an evaluation case with an expected judgment or score established by human experts, domain experts, or another trusted reference process. It gives an LLM judge a concrete quality anchor instead of leaving scoring behavior entirely to a prompt or model preference. section 9.0 section 9.6.1

Use gold examples for two related but distinct evaluation activities:

  • Calibrate the judge by showing what different scores or quality levels look like, such as a score of 5 versus a score of 2. section 9.6.1
  • Validate the judge by comparing its outputs with gold labels, human ratings, or known decision criteria; calibration alone does not establish reliability. section 9.11
  • Inspect disagreements and monitor consistency across repeated runs before trusting the judge for important decisions. section 9.5.4

Movie recommender calibration

A recommender evaluation can include several movie outputs already scored by humans or domain experts. Supplying these gold-standard cases in the judge prompt anchors its scoring and reasoning to the intended quality definition, reducing arbitrary scoring and subjective drift. section 9.6.1

Gold examples support scalable semantic evaluation, but they do not make an LLM judge objective. Humans still define the rubric, choose representative cases, inspect disagreements, and decide whether the judge is reliable enough for the decision. section 9.0 section 9.3

How AI Model Evaluation covers it

How deep it goes
Defines calibration and distinguishes it from validation, but stops short of prescribing a complete dataset construction procedure.
Examples
Uses human- or domain-expert-scored cases, score anchors, and a movie recommender evaluation prompt.
What it assumes
Assumes a defined rubric and trusted human or domain-expert judgments rather than generating evaluation criteria automatically.
What you can do after
Apply it via an LLM-as-a-judge by anchoring scores, comparing outputs with trusted ratings, and inspecting disagreement cases.
How it connects
Connects gold examples to LLM-as-a-judge, calibration, validation, rubrics, human judgment, and domain expertise.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.