A scalar judgment assigns an output a value on a numerical or ordinal scale. It captures degrees of a quality rather than forcing a yes-or-no decision—for example, rating an explanation’s clarity from 1 to 5 or classifying advice as insufficiently, appropriately, or overly cautious. section 5.2
Use scalar judgments when a criterion lies along a spectrum or when tracking gradual changes matters. They can preserve distinctions that a binary label would flatten, such as whether advice is reassuring without becoming alarmist. The choice depends on the evaluation goal and, often, the metrics already selected. section 5.2 section 5.2.1
Account for the trade-offs before choosing a scale:
- Ratings may take more deliberation than binary decisions, especially when adjacent scale points are hard to distinguish. section 5.2.1
- Judges may interpret points differently without clear definitions and examples, and may avoid extreme responses. section 5.2.1
- More granularity does not guarantee better rankings or decisions. section 5.2.1
For a complex quality such as helpfulness or appropriateness, replace a single subjective score with several concrete yes-or-no checks when those criteria can be defined clearly. This can make judgments more reliable and interpretable. A scalar rating may still be useful when gradation itself is important. section 5.2.1 section 5.9
Figure 5.1 Judgments can be binary or scalar, and absolute or relative, producing four distinct ways of assessing examples depending on the evaluation task and decision requirements.

Compare scalar and binary judgments with absolute and relative judgments; these are independent choices that produce four formats. section 5.2
