liveBook · Manning

Relative judgment

As explained in Evaluating AI Systems

A method of assessment in which judges compare outputs with one another rather than assigning each an independent score.

Relative judgment compares two or more outputs directly to decide which is preferable. It can use pairwise comparisons or rank a set of responses, and is useful when evaluating system variants or when quality differences are subtle. section 5.2 section 5.2.2

Use relative judgments with these benefits and limits in mind:

  • Comparisons can be easier and more consistent for judges than assigning absolute scores, and may reveal preferences between outputs that both seem acceptable. section 5.2.2
  • Preferences can become unstable when outputs are very similar. Comparisons also cannot establish whether any output meets a required standard or threshold. section 5.2.2
  • Watch for position bias and contrast bias: the display order or comparison set can shift which output seems better. section 5.2.2
  • Use absolute judgments alongside comparisons when enforcing minimum requirements, such as safety thresholds or policy compliance. section 5.9

Comparing system safety

System A may be judged safer than System B in most comparisons while still producing unsafe outputs. A relative preference alone does not show that either system meets a safety threshold. section 5.2.2

Figure 5.1 Judgments can be binary or scalar, and absolute or relative, producing four distinct ways of assessing examples depending on the evaluation task and decision requirements.

Use the diagram to distinguish relative from absolute judgments and binary from scalar judgments; these choices combine into four assessment formats. section 5.2

How Evaluating AI Systems covers it

How deep it goes
Defines pairwise comparisons and rankings, then weighs cognitive ease against instability, bias, and inability to enforce thresholds.
Examples
Uses system safety comparisons and shows how output position or a contrasting response can skew preferences.
What you can do after
Evaluate system variants and build rankings while retaining absolute checks for standards and minimum thresholds.
How it connects
Connects relative judgments with absolute judgments, pairwise comparisons, rankings, binary judgments, scalar judgments, and rubrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.