Position bias occurs when an LLM judge changes its pairwise verdict because the candidates’ positions change, even though the answers are identical. section 2.2.3
Swap the candidates
A judge picks the grounded answer when it appears on the left, then picks the hallucinated answer when the same answers switch sides. Comparing both orders and requiring agreement filters out this unstable case as a tie. section 2.5 section 2.2.3
Figure 2.5 details the internal process of how LLM-as-Judge uses pairwise evaluation to compare multiple outputs against the same rubric without having position bias modify the answer.

A diagram of pairwise judging with the same rubric helps visualize how answer order can affect comparisons. section 2.2.3





