Relative judgment compares two or more outputs directly to decide which is preferable. It can use pairwise comparisons or rank a set of responses, and is useful when evaluating system variants or when quality differences are subtle. section 5.2 section 5.2.2
Use relative judgments with these benefits and limits in mind:
- Comparisons can be easier and more consistent for judges than assigning absolute scores, and may reveal preferences between outputs that both seem acceptable. section 5.2.2
- Preferences can become unstable when outputs are very similar. Comparisons also cannot establish whether any output meets a required standard or threshold. section 5.2.2
- Watch for position bias and contrast bias: the display order or comparison set can shift which output seems better. section 5.2.2
- Use absolute judgments alongside comparisons when enforcing minimum requirements, such as safety thresholds or policy compliance. section 5.9
Comparing system safety
System A may be judged safer than System B in most comparisons while still producing unsafe outputs. A relative preference alone does not show that either system meets a safety threshold. section 5.2.2
Figure 5.1 Judgments can be binary or scalar, and absolute or relative, producing four distinct ways of assessing examples depending on the evaluation task and decision requirements.

Use the diagram to distinguish relative from absolute judgments and binary from scalar judgments; these choices combine into four assessment formats. section 5.2
