liveBook · Manning

Judgment format

As explained in Evaluating AI Systems

A form in which a judgment is expressed, such as a binary choice, a scalar value, or a comparison between outputs.

When evaluating whether an answer is supported by retrieved evidence, raters might choose a yes-or-no label or assign a score. Judgment format determines how an evaluation records that decision. Metrics often dictate the format, but custom metrics require an explicit choice. section 5.2

Choose along two independent dimensions:

  • Binary or scalar: classify an output as meeting a criterion or not, or rate it on a numerical or ordinal scale. section 5.2
  • Absolute or relative: assess one output against a fixed standard, or compare outputs through pairwise choices or rankings. section 5.2
  • The dimensions can be combined: judgments may be binary or scalar, and absolute or relative. Each combination involves different trade-offs in information, judging effort, and consistency. section 5.2

For LLM judges, format can affect reliability. Binary labels can be a useful starting point when acceptable and unacceptable outputs are clearly distinct. When more nuance is needed, a few well-defined categories may be easier to apply consistently than a broad numeric scale with unclear distinctions between adjacent scores. section 5.6

Figure 5.1 Judgments can be binary or scalar, and absolute or relative, producing four distinct ways of assessing examples depending on the evaluation task and decision requirements.

Use the diagram to compare the two dimensions and their four combinations. section 5.2

How Evaluating AI Systems covers it

How deep it goes
Explains binary versus scalar and absolute versus relative judgments, their combinations, and selected reliability trade-offs.
Examples
Uses evidence support, medical-image classification, advice safety, clinical clarity ratings, pairwise comparisons, and response rankings.
What it assumes
Assumes selected metrics often guide format, while custom metrics require an explicit format decision.
What you can do after
Evaluate outputs by selecting binary or scalar judgments and absolute or relative comparisons suited to the task.
How it connects
Connects judgment format to evaluation metrics, rubrics, rating scales, and human–LLM agreement.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.