A canary set is a fixed evaluation dataset used to detect changes in evaluator behavior or other evaluation-pipeline components. Re-running it against a stored baseline helps distinguish a judge change from a genuine regression in the system being evaluated. section 9.8 section 10.6.2
Use a canary set as part of a repeatable validation and triage loop:
- Select representative cases that reflect the evaluation task, including easy, ambiguous, erroneous, and difficult examples. section 9.8
- Collect human ratings or gold labels, then compare judge outputs with those references before scaling evaluation. section 9.8 section 9.11
- Lock the prompt, rubric, judge model, and baseline results; rerun the fixed set whenever a pipeline component changes. section 9.8 section 10.6.2
- When scores move, check dataset changes, judge versions, prompts, rubrics, and stability before declaring a model regression. section 10.6.2
Recommendation-explanation canary set
Include explanations that are accurate, hallucinated, irrelevant, or persuasive but unsupported by the user’s history. Compare the judge’s results with trusted labels, inspect disagreements, and rerun the same cases after changing the prompt or judge model. section 9.8
A canary set is not a substitute for validation, stability testing, or human judgment. It is a recurring regression signal that makes evaluator drift visible and helps target further investigation. section 10.7 section 9.11
