liveBook · Manning

Position bias

As explained in Retrieval Augmented Generation, The Foundational Ideas

Position bias is an evaluation bias in which a judge favors whichever option appears first in a comparison.

Position bias makes an evaluator favor whichever answer appears first in a pairwise comparison. It is a reliability problem for LLM-as-a-judge pipelines: changing only the order of near-identical answers can change the verdict. section 14.3.3

Order changes the result

In 2023 evaluations, swapping answer order changed Claude-v1's verdict three times out of four and GPT-3.5's about half the time. Current judges are steadier, with a 2026 comparison measuring position bias at 0.04 or below across five judges, although other biases can dominate. section 14.3.3

Use position bias as a diagnostic question when evaluating pairwise judges:

  • Does the verdict change when the same two answers exchange positions? section 14.3.3
  • Which bias currently dominates: order, style, or verbosity? section 14.3.3
  • Could stochastic variance obscure the result even when position bias is low? section 14.3.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines order preference and quantifies it through answer-swapping results, but does not develop a mitigation procedure.
Examples
Uses Claude-v1, GPT-3.5, near-identical answers, and answer-order swaps as concrete evaluation cases.
What it assumes
Assumes pairwise judging and distinguishes order effects from stochastic variation across repeated runs.
What you can do after
Diagnose judge instability by comparing verdicts after swapping answer order.
How it connects
Connects position bias with LLM-as-a-judge evaluation, pairwise comparison, style bias, verbosity bias, and stochastic variance.

Also explained in

Compare how 6 books explain position bias →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.