Position bias occurs when an LLM judge favors a response because of where it appears in the prompt rather than its quality. In a pairwise comparison, a consistent preference for the first response can make a model comparison measure presentation order instead of quality. section 9.7.1
Test for position bias by randomizing response order and running swapped-order evaluations. Compare verdicts by response content, not just by the A or B labels. Frequent winner changes after a swap suggest that the judge or rubric is not stable enough. section 9.7.1 section 10.4.3
For a swapped-order rerun:
- Run the comparison once with the two responses in their original positions. section 10.4.3
- Run the prompt again with the actual response contents exchanged. Keep the template, system instructions, criteria, and output schema identical. section 10.4.3
- Check whether the same response wins in both runs. Relabeling logged results without rerunning the prompt does not test order effects. section 10.4.3
Comparing two model outputs
Place model_v1’s output in the Response A slot and model_v2’s in Response B, using accuracy, relevance, and clarity as criteria. Then rerun with model_v2 in A and model_v1 in B. If the judge chooses A both times, it has selected different response contents; investigate the apparent order sensitivity rather than treating both verdicts as agreement. section 10.4.3
A related position effect can distort recommendation metrics: a movie repeatedly placed first may receive clicks because of its position rather than genuine interest. Internal beta testing can reveal the ranking pattern before launch, though the remedy addresses the recommender rather than an LLM judge. section 7.2





