A checklist item is one grounded, independently traceable criterion that resolves to pass or fail. It makes evaluation failures specific instead of hiding them inside a vague overall judgment. section 10.2.2 section 10.2.3
Write each item so that a reviewer or judge can resolve it consistently:
- Ground it in an observable source or output rather than subjective impressions such as whether a summary feels accurate. section 10.2.2
- Keep it singular: split checks for the effective date and disclosure amount so each failure remains traceable. section 10.2.2
- Make it binary by construction, reserving broad qualities such as writing quality for a scored rubric dimension. section 10.2.2
- Derive it from observed judge disagreements or production incidents, then question items that never catch a real failure. section 10.2.2
A traceable evaluation item
“Does the summary avoid introducing any numeric figure not present in the source document?” is directly checkable and produces a specific failure when an unsupported dollar figure appears. “Is the summary well-written?” is too subjective for a binary checklist item. section 10.2.2
Logging items independently makes debugging targeted. You can identify the exact failed claim, revise an ambiguous item without rewriting the checklist, calculate pass rates per behavior, and locate regressions between model versions. section 10.2.3
