liveBook · Manning

Absolute judgment

As explained in Evaluating AI Systems

A method of assessment in which a judge evaluates an output independently, such as by assigning it a score on a scale.

A system can outperform another and still produce unsafe outputs. Absolute judgment assesses one output on its own against a fixed standard or rubric, helping determine whether it meets a criterion or threshold. section 5.2 section 5.9

Absolute judgment can use either of two formats:

  • Binary: decide whether an output meets a criterion, such as whether advice is safe. section 5.2
  • Scalar: assign a score or category, such as rating an explanation’s clarity from 1 to 5. section 5.2
  • Relative judgment instead compares outputs. It can reveal preferences and subtle differences, but cannot establish that any output meets a required standard. Comparisons can also be affected by position and contrast bias. section 5.2.2

Use absolute judgments when evaluation must enforce a minimum requirement, such as safety or policy compliance. Clear rubrics and observable criteria help make judgments more consistent; relative comparisons can complement, but not replace, threshold checks. section 5.9 section 5.2.2

Figure 5.1 Judgments can be binary or scalar, and absolute or relative, producing four distinct ways of assessing examples depending on the evaluation task and decision requirements.

Shows how binary versus scalar and absolute versus relative choices combine into four judgment formats. section 5.2

How Evaluating AI Systems covers it

How deep it goes
Defines absolute versus relative and binary versus scalar judgments, then outlines their trade-offs without giving a detailed scoring procedure.
Examples
Uses supported claims, tumor detection, chatbot safety, and 1–5 clarity ratings to illustrate judgment formats.
What you can do after
Evaluate whether a single output meets a fixed standard, including a minimum safety or compliance threshold.
How it connects
Connects absolute judgment with binary and scalar judgments, relative judgments, rubrics, and evaluation thresholds.

Also explained in

Compare how 2 books explain absolute judgment →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.