Pairwise comparison
A pairwise comparison is an evaluation of two alternatives against one another to determine their relative preference, value, or rank.
Bayesian Optimization in Action
A pairwise comparison is an observation that indicates which of two candidate inputs has the greater objective or preference value.
- How it is used
- Elicits user preferences for recommendation and preference optimization when exact numerical ratings are noisy or burdensome.
- How deep it goes
- Defines comparisons, encodes them as index pairs, and trains an approximate GP posterior rather than deriving the non-GP posterior.
- Examples
- Uses A/B-style product feedback, car choices, and Hawaiian shirts whose flower count defines a one-dimensional preference search.
- What you can do after
- Apply it via BoTorch to encode preferences, fit a PairwiseGP, and select incumbent-versus-challenger comparisons with acquisition policies.
Read how Bayesian Optimization in Action explains pairwise comparison →
AI Model Evaluation
A pairwise comparison is an evaluation procedure that compares two alternatives against shared criteria to determine which is preferable or whether they are tied.
- How it is used
- Compares two outputs to choose between model variants, prompt revisions, or candidate responses on defined criteria.
- How deep it goes
- Explains task definition, prompt structure, output schema, cost trade-offs, validation, and common judge biases.
- Examples
- Uses two generated summaries judged for accuracy, relevance, clarity, and overall preference.
- What you can do after
- Apply it via an LLM judge to compare candidate outputs and validate whether its verdicts support the intended decision.
What to know first
- relevanceRelevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- JSONJSON is a language-independent text-based data-interchange format that represents structured values using objects, arrays, strings, numbers, booleans, and null.
Related concepts
- ranking scorecontrasts with · A ranking score is a numerical evaluation of an item or outcome used to compare candidates or determine their relative order.
- swapped-order evaluationused by · A swapped-order evaluation repeats a comparison after reversing the presentation order of the alternatives to detect order-dependent judgments.
- preferenceuses · A preference is a comparative judgment or inclination that favors one alternative, outcome, or characteristic over others.
- LLM-as-a-judgeuses · LLM-as-a-judge is an evaluation technique in which a language model assesses or compares outputs according to specified criteria.
- preference learning problemused by · A preference learning problem is the task of inferring or optimizing desired behavior from comparative judgments about alternative outputs or actions.
Questions about pairwise comparison
What is pairwise comparison?
A pairwise comparison is an evaluation of two alternatives against one another to determine their relative preference, value, or rank.
Which Manning books explain pairwise comparison?
Bayesian Optimization in Action and AI Model Evaluation.
What should I know before learning pairwise comparison?
Manning books that teach pairwise comparison usually introduce relevance, accuracy and JSON first.
What is the difference between pairwise comparison and ranking score?
A pairwise comparison is an evaluation of two alternatives against one another to determine their relative preference, value, or rank. A ranking score is a numerical evaluation of an item or outcome used to compare candidates or determine their relative order.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

