liveBook · Manning

Position bias

As explained in AI Model Evaluation

Position bias is the tendency for a response's displayed position in an evaluation prompt to affect the judge's verdict independently of the response content.

Position bias occurs when an LLM judge favors a response because of where it appears in the prompt rather than its quality. In a pairwise comparison, a consistent preference for the first response can make a model comparison measure presentation order instead of quality. section 9.7.1

Test for position bias by randomizing response order and running swapped-order evaluations. Compare verdicts by response content, not just by the A or B labels. Frequent winner changes after a swap suggest that the judge or rubric is not stable enough. section 9.7.1 section 10.4.3

For a swapped-order rerun:

  • Run the comparison once with the two responses in their original positions. section 10.4.3
  • Run the prompt again with the actual response contents exchanged. Keep the template, system instructions, criteria, and output schema identical. section 10.4.3
  • Check whether the same response wins in both runs. Relabeling logged results without rerunning the prompt does not test order effects. section 10.4.3

Comparing two model outputs

Place model_v1’s output in the Response A slot and model_v2’s in Response B, using accuracy, relevance, and clarity as criteria. Then rerun with model_v2 in A and model_v1 in B. If the judge chooses A both times, it has selected different response contents; investigate the apparent order sensitivity rather than treating both verdicts as agreement. section 10.4.3

A related position effect can distort recommendation metrics: a movie repeatedly placed first may receive clicks because of its position rather than genuine interest. Internal beta testing can reveal the ranking pattern before launch, though the remedy addresses the recommender rather than an LLM judge. section 7.2

How AI Model Evaluation covers it

How deep it goes
Defines the bias and details controlled swapped-order reruns, including the need to swap response content rather than just relabel results.
Examples
Uses two model outputs compared against accuracy, relevance, and clarity, plus a repeatedly top-ranked movie in internal beta testing.
What you can do after
Diagnose order sensitivity by rerunning pairwise judgments with response contents swapped and comparing the winners.
How it connects
Connects position bias to pairwise comparison, LLM-as-a-judge, rubric stability, and recommendation ranking.

Also explained in

Compare how 6 books explain position bias →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.