Position bias is an evaluator’s tendency to favor a response because of its placement or presentation order rather than its intrinsic quality. In pairwise LLM judging, the first response may receive a primacy advantage, while some judges favor the last response through recency bias. section 4.3.1
Swapping the responses
Present Response A first and Response B second, then reverse their positions without changing their content. If the judge selects A in the first run but B after the swap, the verdict depends on order rather than quality. section 4.3.1
Reduce position bias with progressively stronger checks:
- Run each pairwise comparison twice with the response positions swapped, and treat agreement as the condition for a confident verdict. section 4.3.1
- Reserve swapped evaluations for close comparisons when evaluation cost matters; use a single pass when one response is clearly superior. section 4.3.1
- Add few-shot examples and explicit instructions requiring equal consideration of both positions. section 4.3.1
- Aggregate judgments from multiple models to reduce model-specific positional preferences, then calibrate results against human preferences. section 4.3.1 section 4.3.5





