An LLM judge evaluates another system’s outputs by applying a rubric in an evaluation prompt and returning scores or pass/fail decisions. It scales repeated agent validation as prompts, models, and data change, while complementing code-based checks and human review. section 10.3
Two common judging patterns are:
- Single-output grading scores one answer against a rubric. Including a reference answer can improve consistency, but preparing references takes additional effort. section 10.3.1
- Pairwise comparison asks which of two answers better meets the criteria, or whether they tie. Relative judgments suit subjective qualities and A/B testing; results can be aggregated into win rates. section 10.3.1
Applying a rubric
For a research-paper summary, a judge can check whether the answer captures key findings and is concise and accurate, then return PASS or FAIL with a brief explanation. For renewable-energy answers, it can compare two candidates and choose A, B, or Tie. section 10.3.1
