liveBook · Manning

Rubric

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An evaluation rubric is a structured set of explicit criteria and guidance for assigning judgments to an AI system’s outputs.

An evaluation rubric turns a vague quality judgment into explicit, observable criteria for assigning consistent scores to AI outputs. Precise rubrics improve reproducibility and agreement with human judgment; vague labels produce inconsistent results. section 4.4.3

Build an effective rubric from four components:

  • Define the quality dimension concretely, distinguish it from other dimensions, and identify which response features to examine. section 4.4.3
  • Specify observable criteria for every score level instead of relying on comparative terms such as “better” or “worse.” section 4.4.3
  • Add representative sample responses across the scale, including ambiguous middle cases, to provide calibration anchors. section 4.4.3
  • Document edge-case handling so evaluations remain consistent when responses fall outside ordinary examples. section 4.4.3

Helpfulness rubric

Measure whether a response directly addresses the user’s need with accurate, actionable information. Score 1 for an unrelated or useless response; 3 for useful but incomplete or slightly inaccurate information; 5 for a response that exceeds expectations by anticipating follow-up needs, adding context, or offering multiple approaches. section 4.4.3

Rubrics support both single-response scoring and pairwise preference judgments. Absolute scoring needs especially clear definitions, while pairwise comparison often provides a more reliable relative judgment. section 4.2.2

Treat the rubric as a monitored configuration: calibrate judges against human-labeled data, inspect score distributions, run golden-set regression tests, and sample evaluations for human verification. section 4.5 section 7.5.8

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Descends from quality definitions to score-level criteria, calibration examples, edge-case guidance, and ongoing reliability monitoring.
Examples
Uses helpfulness score levels, coherence criteria for summaries, representative response examples, and ambiguous middle cases.
What you can do after
Evaluate open-ended outputs with reproducible criteria and calibrate judges against human-labeled comparisons.
How it connects
Connects evaluation rubrics with LLM-as-a-judge systems, human judgment, pairwise preference evaluation, calibration, and RLAIF.

Also explained in

Compare how 7 books explain rubric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.