liveBook · Manning

Swapped-order evaluation

As explained in AI Model Evaluation

A swapped-order rerun is a repeated pairwise evaluation in which the same two responses are placed in opposite prompt positions to test whether the judgment changes with ordering.

Swapped-order evaluation reruns the same pairwise judgment with two responses in opposite prompt positions. It tests whether the verdict changes because of position rather than response quality. section 9.7.1 section 10.4.3

Run the check without changing the evaluation itself:

  • Keep the system instructions, prompt template, criteria, and output schema identical. section 10.4.3
  • Run the comparison once as Response A versus Response B, then again as Response B versus Response A. section 10.4.3
  • Compare the two verdicts; frequent winner changes indicate position sensitivity or an insufficiently stable judge. section 9.7.1
  • Swap the response content before rerunning; relabeling logged results afterward does not test ordering bias. section 10.4.3

Model comparison

Evaluate model_v1 as Response A and model_v2 as Response B using criteria of accuracy, relevance, and clarity, then rerun the identical JSON-producing prompt with their positions reversed. Compare the returned winners. section 10.4.3

Use swapped-order reruns alongside repeated sampling in recurring validation. Repeated sampling exposes general variance; swapped ordering targets position bias specifically. section 10.7

How AI Model Evaluation covers it

How deep it goes
Defines the rerun procedure, fixes the prompt configuration, requires actual content swaps, and interprets verdict changes as a stability signal.
Examples
Uses reversed Response A and Response B placements for model_v1 and model_v2 under accuracy, relevance, and clarity criteria.
What it assumes
Assumes pairwise judging and prompt templates, while clarifying the distinction between actual swapping and post hoc relabeling.
What you can do after
Diagnose position bias by rerunning pairwise prompts and comparing verdict changes across response orders.
How it connects
Connects position bias, pairwise evaluation, repeated sampling, judge stability, and validation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.