Swap-and-agree reduces position bias in pairwise LLM judging: compare the same answers in both orders, keep a winner only when the judgments agree, and return TIE when they disagree. The signal includes the winner, an agreement flag, and a one-sentence reason. section 2.2.3
A judge that changes its mind
For a question about the 2017 Attention paper, one answer correctly describes the Transformer and another falsely attributes BERT to the paper. If swapping their positions changes the judge’s pick, swap-and-agree returns TIE rather than trusting either verdict. section 2.2.3 section 2.5
Figure 2.5 details the internal process of how LLM-as-Judge uses pairwise evaluation to compare multiple outputs against the same rubric without having position bias modify the answer.

Shows how pairwise evaluation compares outputs under the same rubric while checking for position bias. section 2.2.3
