liveBook · Manning

Judge-based signal

As explained in Self-Improving Agents

Feedback produced when a language model evaluates an agent’s output.

When labeled answers are expensive or a task has no clean execution outcome, a judge-based signal asks a language model to evaluate an agent’s output. Because it requires another model call, it is available for many tasks. section 1.4.1

Common forms differ in what the judge is asked to do:

  • Absolute scoring rates one output on a scale. section 1.4.1
  • Pairwise preference chooses the better of two outputs. section 1.4.1
  • Contrastive judging identifies what one output does better or worse than another. section 1.4.1

Availability comes with noise: judges can be inconsistent, favor longer or more confident answers, and be confidently wrong. Pairwise and contrastive judgments are generally steadier than absolute scores. Treat the signal as fallible feedback, not ground truth. section 1.4.1

Figure 1.9 The signal family arranged by precision against availability. Ground-truth and environment signals are the most precise but the rarest; judge-based signals are available almost everywhere but are noisier, and process and reflection signals assess how the agent worked rather than only what it produced.

Compares signal types by precision and availability, placing judge-based signals among the widely available but noisier options. section 1.4.1

How Self-Improving Agents covers it

How deep it goes
Defines three judging formats and their reliability trade-offs, but gives no implementation procedure.
Examples
Compares two outputs through pairwise preference or asks what one does better or worse through contrastive judging.
What you can do after
Evaluate agent outputs with model judgments while accounting for inconsistency, bias, and possible error.
How it connects
Connects to ground-truth and environment signals, process and reflection signals, and self-improvement loops.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.