liveBook · Manning

Swap-and-agree

As explained in Self-Improving Agents

A pairwise judging technique that evaluates answers in both possible orders and accepts a verdict only when the judgments agree.

Swap-and-agree reduces position bias in pairwise LLM judging: compare the same answers in both orders, keep a winner only when the judgments agree, and return TIE when they disagree. The signal includes the winner, an agreement flag, and a one-sentence reason. section 2.2.3

A judge that changes its mind

For a question about the 2017 Attention paper, one answer correctly describes the Transformer and another falsely attributes BERT to the paper. If swapping their positions changes the judge’s pick, swap-and-agree returns TIE rather than trusting either verdict. section 2.2.3 section 2.5

Figure 2.5 details the internal process of how LLM-as-Judge uses pairwise evaluation to compare multiple outputs against the same rubric without having position bias modify the answer.

Shows how pairwise evaluation compares outputs under the same rubric while checking for position bias. section 2.2.3

How Self-Improving Agents covers it

How deep it goes
Explains reversed pairwise calls, agreement gating, and a LEFT/RIGHT/TIE signal with a reason.
Examples
Uses Transformer-versus-BERT answer judgments and an exercise that removes the reverse call to expose position bias.
What it assumes
Requires two candidate answers and a judge that can return a structured winner and reason.
What you can do after
Diagnose position-biased verdicts by swapping answer order and treating disagreements as ties.
How it connects
Links pairwise judging to position bias, evaluation rubrics, and LLM-as-a-Judge.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Improving Agents. The section links point to the parts of the book it draws on.