Position bias makes an evaluator favor whichever answer appears first in a pairwise comparison. It is a reliability problem for LLM-as-a-judge pipelines: changing only the order of near-identical answers can change the verdict. section 14.3.3
Order changes the result
In 2023 evaluations, swapping answer order changed Claude-v1's verdict three times out of four and GPT-3.5's about half the time. Current judges are steadier, with a 2026 comparison measuring position bias at 0.04 or below across five judges, although other biases can dominate. section 14.3.3
Use position bias as a diagnostic question when evaluating pairwise judges:
- Does the verdict change when the same two answers exchange positions? section 14.3.3
- Which bias currently dominates: order, style, or verbosity? section 14.3.3
- Could stochastic variance obscure the result even when position bias is low? section 14.3.3





