liveBook · Manning

Binary judgment

As explained in Evaluating AI Systems

A judgment that classifies an output as either meeting a criterion or not, with no intermediate states.

A binary judgment classifies an output as either meeting a criterion or not, with no intermediate states. In evaluation, it can answer whether a claim is supported by retrieved documents or whether a chatbot’s advice is safe. The judgment format is often dictated by the chosen metric, but may need to be designed explicitly for a custom metric. section 5.2

Binary judgments work best when the criterion is clear-cut and borderline cases are rare—for example, whether a medical chatbot mentions a specific drug contraindication. They are less suitable for qualities such as clarity or reassuring tone, where forcing a yes/no answer can hide meaningful differences. Scalar judgments preserve gradations, but may require more deliberation and introduce inconsistent interpretations of scale points. section 5.2.1

Binary and scalar describe the form of a judgment; absolute and relative describe what is being assessed. An absolute judgment evaluates an output against a standard, while a relative judgment compares outputs. These choices are independent, so a binary judgment can be either absolute or relative. section 5.2

Figure 5.1 Judgments can be binary or scalar, and absolute or relative, producing four distinct ways of assessing examples depending on the evaluation task and decision requirements.

The figure maps binary and scalar judgments against absolute and relative judgments. section 5.2

When a broad quality needs more nuance, one option is to replace a single subjective rating with several well-defined yes/no checks. This requires understanding the construct and choosing concrete criteria; rubrics should define observable signals and include examples and counterexamples. Multiple checks can often make judgments more reliable and interpretable without relying on a broad numeric scale. section 5.2.1 section 5.9

Judging retrieved support

For an LLM judge, a binary supported-or-unsupported decision can be a starting point when the distinction is clear. If partial support matters, a small set of defined labels—fully supported, partially supported, unsupported, and contradicted—can capture distinctions that a binary decision would miss. section 5.6

How Evaluating AI Systems covers it

How deep it goes
Explains format selection and trade-offs, including decomposition of broad qualities into concrete yes/no checks.
Examples
Illustrates judgments with retrieved-document support, drug contraindications, emergency symptoms, and chatbot advice.
What you can do after
Apply it by selecting clear-cut criteria and decomposing broader evaluation qualities into well-defined yes/no checks.
How it connects
Connects binary judgments to scalar judgments, absolute and relative judgments, rubrics, and LLM judges.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.