Swapped-order evaluation reruns the same pairwise judgment with two responses in opposite prompt positions. It tests whether the verdict changes because of position rather than response quality. section 9.7.1 section 10.4.3
Run the check without changing the evaluation itself:
- Keep the system instructions, prompt template, criteria, and output schema identical. section 10.4.3
- Run the comparison once as Response A versus Response B, then again as Response B versus Response A. section 10.4.3
- Compare the two verdicts; frequent winner changes indicate position sensitivity or an insufficiently stable judge. section 9.7.1
- Swap the response content before rerunning; relabeling logged results afterward does not test ordering bias. section 10.4.3
Model comparison
Evaluate model_v1 as Response A and model_v2 as Response B using criteria of accuracy, relevance, and clarity, then rerun the identical JSON-producing prompt with their positions reversed. Compare the returned winners. section 10.4.3
Use swapped-order reruns alongside repeated sampling in recurring validation. Repeated sampling exposes general variance; swapped ordering targets position bias specifically. section 10.7
