A correct answer can still miss what the user needs most. If a support agent answers a yes-or-no return question accurately but buries the verdict, a second model can score the answer against a plain-language rubric: does it answer the question directly? That second model is an LLM-as-judge. It can score one output or compare candidates to pick a winner. section 1.2.1
Turn a judgment into a prompt improvement
A support agent gives three accurate sentences about late returns, but the “yes” appears only at the end. The judge identifies that gap. A search proposes prompt variants, and the judge prefers an instruction to begin with a direct yes or no, then explain. A deploy gate sits between the winning prompt and the running agent; the model’s weights do not change. section 1.2.1
The rubric defines what “good” means for the task. A comparison can supply a question, a reference answer, two candidate answers, and instructions that prioritize factual accuracy, grounding in retrieved sources, then clarity and directness. The resulting verdict is a quality signal, not ground truth: an extra model call makes it readily available, but a model’s opinion is less precise than a reliable reference or human review. section 2.2.1 section 2.2.3
For pairwise judgments, guard against answer-order bias with swap-and-agree:
- Judge A against B, then repeat with their left and right positions reversed. section 2.2.3
- Keep the winner only if both runs select the same underlying answer. Otherwise return TIE rather than trusting a position-sensitive verdict. section 2.2.3
- Return the winner or TIE, an agreement flag, and a short reason as the measurement used by the improvement loop. section 2.2.3
Figure 2.5 details the internal process of how LLM-as-Judge uses pairwise evaluation to compare multiple outputs against the same rubric without having position bias modify the answer.

The diagram traces two answer orders through the judge and shows where agreement determines whether a winner is retained. section 2.2.3
Test the judge as carefully as the agent. Swapping identical candidates can expose position bias; changing the rubric’s criterion order can change a close verdict. Use the signal to compare prompt variants, but keep a deployment gate before promoting a winner. section 2.5 section 1.2.1




















