liveBook · Manning

Position bias

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Position bias is a tendency for an evaluator to favor an answer based on its position or presentation order rather than its intrinsic quality.

Position bias is an evaluator’s tendency to favor a response because of its placement or presentation order rather than its intrinsic quality. In pairwise LLM judging, the first response may receive a primacy advantage, while some judges favor the last response through recency bias. section 4.3.1

Swapping the responses

Present Response A first and Response B second, then reverse their positions without changing their content. If the judge selects A in the first run but B after the swap, the verdict depends on order rather than quality. section 4.3.1

Reduce position bias with progressively stronger checks:

  • Run each pairwise comparison twice with the response positions swapped, and treat agreement as the condition for a confident verdict. section 4.3.1
  • Reserve swapped evaluations for close comparisons when evaluation cost matters; use a single pass when one response is clearly superior. section 4.3.1
  • Add few-shot examples and explicit instructions requiring equal consideration of both positions. section 4.3.1
  • Aggregate judgments from multiple models to reduce model-specific positional preferences, then calibrate results against human preferences. section 4.3.1 section 4.3.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains primacy and recency effects, demonstrates a perturbation test, and gives mitigation strategies while stopping short of causal certainty.
Examples
Uses two chatbot responses whose order is swapped to reveal whether the judge changes its preferred winner.
What it assumes
Assumes pairwise response comparison and LLM judging, while teaching positional effects and their evaluation safeguards from the basic failure mode.
What you can do after
Diagnose order sensitivity, apply position-swapped comparisons, and interpret judge agreement as a reliability signal rather than ground truth.
How it connects
Connects pairwise comparison with perturbation testing, verbosity bias, self-enhancement bias, calibration, and human oversight.

Also explained in

Compare how 6 books explain position bias →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.