liveBook · Manning

Checklist item

As explained in AI Model Evaluation

A checklist item is a single, grounded, independently traceable criterion in a checklist that can be resolved as pass or fail.

A checklist item is one grounded, independently traceable criterion that resolves to pass or fail. It makes evaluation failures specific instead of hiding them inside a vague overall judgment. section 10.2.2 section 10.2.3

Write each item so that a reviewer or judge can resolve it consistently:

  • Ground it in an observable source or output rather than subjective impressions such as whether a summary feels accurate. section 10.2.2
  • Keep it singular: split checks for the effective date and disclosure amount so each failure remains traceable. section 10.2.2
  • Make it binary by construction, reserving broad qualities such as writing quality for a scored rubric dimension. section 10.2.2
  • Derive it from observed judge disagreements or production incidents, then question items that never catch a real failure. section 10.2.2

A traceable evaluation item

“Does the summary avoid introducing any numeric figure not present in the source document?” is directly checkable and produces a specific failure when an unsupported dollar figure appears. “Is the summary well-written?” is too subjective for a binary checklist item. section 10.2.2

Logging items independently makes debugging targeted. You can identify the exact failed claim, revise an ambiguous item without rewriting the checklist, calculate pass rates per behavior, and locate regressions between model versions. section 10.2.3

How AI Model Evaluation covers it

How deep it goes
Defines grounding, singularity, binary resolution, and failure provenance, then connects item logging to debugging and regression analysis.
Examples
Uses policy dates, disclosure amounts, unsupported figures, attribution, and disclosure-presence checks as concrete criteria.
What it assumes
Assumes an evaluated output, a source for grounding, and comparison signals such as human labels or judge disagreements.
What you can do after
Diagnose/debug with it: isolate judge disagreements, revise ambiguous criteria, compute item pass rates, and detect behavior-specific regressions.
How it connects
Connects checklist items to rubrics, disagreement analysis, human labels, production incidents, prompt iteration, and regression testing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.