When labeled answers are expensive or a task has no clean execution outcome, a judge-based signal asks a language model to evaluate an agent’s output. Because it requires another model call, it is available for many tasks. section 1.4.1
Common forms differ in what the judge is asked to do:
- Absolute scoring rates one output on a scale. section 1.4.1
- Pairwise preference chooses the better of two outputs. section 1.4.1
- Contrastive judging identifies what one output does better or worse than another. section 1.4.1
Availability comes with noise: judges can be inconsistent, favor longer or more confident answers, and be confidently wrong. Pairwise and contrastive judgments are generally steadier than absolute scores. Treat the signal as fallible feedback, not ground truth. section 1.4.1
Figure 1.9 The signal family arranged by precision against availability. Ground-truth and environment signals are the most precise but the rarest; judge-based signals are available almost everywhere but are noisier, and process and reflection signals assess how the agent worked rather than only what it produced.

Compares signal types by precision and availability, placing judge-based signals among the widely available but noisier options. section 1.4.1
