liveBook · Manning

LLM-as-a-judge

As explained in Self-Improving Agents

A pattern in which a large language model evaluates outputs against a rubric, potentially comparing candidates to select the best.

A correct answer can still miss what the user needs most. If a support agent answers a yes-or-no return question accurately but buries the verdict, a second model can score the answer against a plain-language rubric: does it answer the question directly? That second model is an LLM-as-judge. It can score one output or compare candidates to pick a winner. section 1.2.1

Turn a judgment into a prompt improvement

A support agent gives three accurate sentences about late returns, but the “yes” appears only at the end. The judge identifies that gap. A search proposes prompt variants, and the judge prefers an instruction to begin with a direct yes or no, then explain. A deploy gate sits between the winning prompt and the running agent; the model’s weights do not change. section 1.2.1

The rubric defines what “good” means for the task. A comparison can supply a question, a reference answer, two candidate answers, and instructions that prioritize factual accuracy, grounding in retrieved sources, then clarity and directness. The resulting verdict is a quality signal, not ground truth: an extra model call makes it readily available, but a model’s opinion is less precise than a reliable reference or human review. section 2.2.1 section 2.2.3

For pairwise judgments, guard against answer-order bias with swap-and-agree:

  • Judge A against B, then repeat with their left and right positions reversed. section 2.2.3
  • Keep the winner only if both runs select the same underlying answer. Otherwise return TIE rather than trusting a position-sensitive verdict. section 2.2.3
  • Return the winner or TIE, an agreement flag, and a short reason as the measurement used by the improvement loop. section 2.2.3
Figure 2.5 details the internal process of how LLM-as-Judge uses pairwise evaluation to compare multiple outputs against the same rubric without having position bias modify the answer.

The diagram traces two answer orders through the judge and shows where agreement determines whether a winner is retained. section 2.2.3

Test the judge as carefully as the agent. Swapping identical candidates can expose position bias; changing the rubric’s criterion order can change a close verdict. Use the signal to compare prompt variants, but keep a deployment gate before promoting a winner. section 2.5 section 1.2.1

How Self-Improving Agents covers it

How deep it goes
Works through pairwise inputs, ordered rubric criteria, swap-and-agree, and disagreement handling; stops short of treating judge verdicts as ground truth.
Examples
Uses a late-return support answer and a grounded-versus-hallucinated research answer, plus exercises that swap positions and reorder rubric criteria.
What it assumes
Introduces agent construction with Helix and a provided harness before assembling the judge-based improvement loop.
What you can do after
Build it from scratch: implement a pairwise judge with swap-and-agree, then use its verdict to select offline prompt variants.
How it connects
Connects rubrics and gold sets to pairwise evaluation, position bias, prompt optimization, and deployment gates.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.