A system can outperform another and still produce unsafe outputs. Absolute judgment assesses one output on its own against a fixed standard or rubric, helping determine whether it meets a criterion or threshold. section 5.2 section 5.9
Absolute judgment can use either of two formats:
- Binary: decide whether an output meets a criterion, such as whether advice is safe. section 5.2
- Scalar: assign a score or category, such as rating an explanation’s clarity from 1 to 5. section 5.2
- Relative judgment instead compares outputs. It can reveal preferences and subtle differences, but cannot establish that any output meets a required standard. Comparisons can also be affected by position and contrast bias. section 5.2.2
Use absolute judgments when evaluation must enforce a minimum requirement, such as safety or policy compliance. Clear rubrics and observable criteria help make judgments more consistent; relative comparisons can complement, but not replace, threshold checks. section 5.9 section 5.2.2
Figure 5.1 Judgments can be binary or scalar, and absolute or relative, producing four distinct ways of assessing examples depending on the evaluation task and decision requirements.

Shows how binary versus scalar and absolute versus relative choices combine into four judgment formats. section 5.2

