Pairwise comparison evaluates two alternatives against shared criteria and decides whether A, B, or neither is preferable. In LLM evaluation, it is useful when a direct choice matters more than an absolute score: compare a new model variant with a baseline, test a prompt revision, or select between candidate outputs. section 9.11 section 9.5.3
Structure a reliable comparison around four elements:
- Define the decision and criteria before presenting the alternatives. Accuracy, relevance, and clarity give the judge a bounded task instead of the vague instruction to judge an answer. section 9.5.1
- Provide the shared input and both responses, then require separate judgments for each criterion before an overall verdict. section 9.5.1
- Return a constrained result such as A, B, or tie, with reasons in a defined JSON schema. Treat the prompt as a controlled interface that may require iteration. section 9.5.1
- Validate the judge against human ratings, gold labels, known examples, or canary sets; monitor position, verbosity, style, context, and prompt-injection biases. section 9.11
Comparing two generated summaries
Give the judge the user query, Response A, and Response B. Ask which response wins on accuracy, relevance, and clarity, without favoring length or style. Record a winner or tie for every criterion and an overall verdict with justification. This supports a concrete decision such as whether a revised prompt beats the baseline, but many pairwise calls can increase evaluation cost. section 9.5.1 section 9.5.3
Use pairwise comparison deliberately rather than as a default replacement for scoring or ranking. Scoring tracks aggregate trends, while ranking orders several candidates; pairwise comparison offers a direct choice but becomes expensive across many alternatives. Prefer deterministic checks when exact matching, rules, labels, or trusted ground truth are available, and retain human or domain-specific review for high-impact decisions. section 9.5.3 section 9.11

