liveBook · Manning

Position bias

As explained in Self-Improving Agents

A systematic influence of an answer's position in a comparison on a judge's evaluation, independent of answer quality.

Position bias occurs when an LLM judge changes its pairwise verdict because the candidates’ positions change, even though the answers are identical. section 2.2.3

Swap the candidates

A judge picks the grounded answer when it appears on the left, then picks the hallucinated answer when the same answers switch sides. Comparing both orders and requiring agreement filters out this unstable case as a tie. section 2.5 section 2.2.3

Figure 2.5 details the internal process of how LLM-as-Judge uses pairwise evaluation to compare multiple outputs against the same rubric without having position bias modify the answer.

A diagram of pairwise judging with the same rubric helps visualize how answer order can affect comparisons. section 2.2.3

How Self-Improving Agents covers it

How deep it goes
Demonstrates verdict reversal and swap-and-agree filtering, but does not quantify bias or explain its underlying cause.
Examples
Swaps grounded and hallucinated answers, then checks whether the judge reverses its verdict.
What you can do after
Diagnose order-sensitive judgments by swapping candidate positions and requiring agreement across both verdicts.
How it connects
Connects pairwise evaluation with rubrics and agent self-improvement loops.

Also explained in

Compare how 6 books explain position bias →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.