A checklist evaluates a response through atomic, independently verifiable pass/fail checks rather than one holistic score. Each result identifies a specific behavior, making failures easier to interpret and act on. An aggregate score can support dashboards, but the item-level results remain the primary artifact. section 10.2
Use a checklist when failures can be enumerated, partial credit could conceal a serious error, or results need to guide product and engineering fixes. Holistic qualities such as tone and narrative flow may fit a scored rubric better; both formats can be used in the same judge call. section 10.2.1
Write each check so its result is actionable:
- Ground it in something verifiable, such as a source document, rather than asking whether an answer feels accurate. section 10.2.2
- Check one thing at a time and make the outcome clearly pass or fail. Split a combined date-and-amount check into separate items. section 10.2.2
- Derive items from disagreement analysis and production incidents. Reconsider items that never catch a real failure in the validation set. section 10.2.2
Checking a policy summary
Compare a generated summary with its source document. Check the effective date, unsupported figures, attribution, and required disclosures separately. If only the unsupported-figures check fails because the summary cites a $2M figure absent from the source, the result is three passes out of four—with the failed item and evidence preserved. section 10.2
Log results per item to localize disagreements, revise ambiguous checks, and track failure patterns across an evaluation set. Item-level pass rates also make regression testing more precise: a drop in the unsupported-figures pass rate can be distinguished from stable performance on other checks. section 10.2.3
A checklist can also give a lower-cost judge a more concrete task and provide item-level drift signals. It does not eliminate confident errors: audit a random sample of high-confidence verdicts against a stronger judge or human reviewer. section 10.5.2




