When evaluating whether an answer is supported by retrieved evidence, raters might choose a yes-or-no label or assign a score. Judgment format determines how an evaluation records that decision. Metrics often dictate the format, but custom metrics require an explicit choice. section 5.2
Choose along two independent dimensions:
- Binary or scalar: classify an output as meeting a criterion or not, or rate it on a numerical or ordinal scale. section 5.2
- Absolute or relative: assess one output against a fixed standard, or compare outputs through pairwise choices or rankings. section 5.2
- The dimensions can be combined: judgments may be binary or scalar, and absolute or relative. Each combination involves different trade-offs in information, judging effort, and consistency. section 5.2
For LLM judges, format can affect reliability. Binary labels can be a useful starting point when acceptable and unacceptable outputs are clearly distinct. When more nuance is needed, a few well-defined categories may be easier to apply consistently than a broad numeric scale with unclear distinctions between adjacent scores. section 5.6
Figure 5.1 Judgments can be binary or scalar, and absolute or relative, producing four distinct ways of assessing examples depending on the evaluation task and decision requirements.

Use the diagram to compare the two dimensions and their four combinations. section 5.2
