An evaluation rubric turns a vague quality judgment into explicit, observable criteria for assigning consistent scores to AI outputs. Precise rubrics improve reproducibility and agreement with human judgment; vague labels produce inconsistent results. section 4.4.3
Build an effective rubric from four components:
- Define the quality dimension concretely, distinguish it from other dimensions, and identify which response features to examine. section 4.4.3
- Specify observable criteria for every score level instead of relying on comparative terms such as “better” or “worse.” section 4.4.3
- Add representative sample responses across the scale, including ambiguous middle cases, to provide calibration anchors. section 4.4.3
- Document edge-case handling so evaluations remain consistent when responses fall outside ordinary examples. section 4.4.3
Helpfulness rubric
Measure whether a response directly addresses the user’s need with accurate, actionable information. Score 1 for an unrelated or useless response; 3 for useful but incomplete or slightly inaccurate information; 5 for a response that exceeds expectations by anticipating follow-up needs, adding context, or offering multiple approaches. section 4.4.3
Rubrics support both single-response scoring and pairwise preference judgments. Absolute scoring needs especially clear definitions, while pairwise comparison often provides a more reliable relative judgment. section 4.2.2
Treat the rubric as a monitored configuration: calibrate judges against human-labeled data, inspect score distributions, run golden-set regression tests, and sample evaluations for human verification. section 4.5 section 7.5.8






