liveBook · Manning

Pairwise comparison

As explained in AI Model Evaluation

A pairwise comparison is an evaluation procedure that compares two alternatives against shared criteria to determine which is preferable or whether they are tied.

Pairwise comparison evaluates two alternatives against shared criteria and decides whether A, B, or neither is preferable. In LLM evaluation, it is useful when a direct choice matters more than an absolute score: compare a new model variant with a baseline, test a prompt revision, or select between candidate outputs. section 9.11 section 9.5.3

Structure a reliable comparison around four elements:

  • Define the decision and criteria before presenting the alternatives. Accuracy, relevance, and clarity give the judge a bounded task instead of the vague instruction to judge an answer. section 9.5.1
  • Provide the shared input and both responses, then require separate judgments for each criterion before an overall verdict. section 9.5.1
  • Return a constrained result such as A, B, or tie, with reasons in a defined JSON schema. Treat the prompt as a controlled interface that may require iteration. section 9.5.1
  • Validate the judge against human ratings, gold labels, known examples, or canary sets; monitor position, verbosity, style, context, and prompt-injection biases. section 9.11

Comparing two generated summaries

Give the judge the user query, Response A, and Response B. Ask which response wins on accuracy, relevance, and clarity, without favoring length or style. Record a winner or tie for every criterion and an overall verdict with justification. This supports a concrete decision such as whether a revised prompt beats the baseline, but many pairwise calls can increase evaluation cost. section 9.5.1 section 9.5.3

Use pairwise comparison deliberately rather than as a default replacement for scoring or ranking. Scoring tracks aggregate trends, while ranking orders several candidates; pairwise comparison offers a direct choice but becomes expensive across many alternatives. Prefer deterministic checks when exact matching, rules, labels, or trusted ground truth are available, and retain human or domain-specific review for high-impact decisions. section 9.5.3 section 9.11

How AI Model Evaluation covers it

How deep it goes
Explains task definition, prompt structure, output schema, cost trade-offs, validation, and common judge biases.
Examples
Uses two generated summaries judged for accuracy, relevance, clarity, and overall preference.
What it assumes
Defines quality criteria, input context, and decision goals before comparing outputs; assumes no fixed scoring scale.
What you can do after
Apply it via an LLM judge to compare candidate outputs and validate whether its verdicts support the intended decision.
How it connects
Connects pairwise comparison with scoring, ranking, prompt design, validation, and LLM-as-a-judge.

Also explained in

Compare how 2 books explain pairwise comparison →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.