Disagreement review investigates cases where an LLM judge and human labels, gold examples, or another judge reach different conclusions. It treats each mismatch as a debugging signal rather than automatically trusting a fluent explanation or score. section 9.7.6 section 9.8
Review each disagreement by asking:
- Did the judge apply the rubric correctly? section 9.7.6
- Did it use the right evidence, or reward irrelevant preferences such as verbosity? section 9.7.6 section 9.8
- Did missing context or an ambiguous rubric cause the mismatch? section 9.8
- Can the failure be localized to one checklist criterion rather than an overall score? section 10.2.3
Localizing a summary disagreement
A human reviewer marks a summary accurate, but the judge fails it. A checklist may reveal that only “no unsupported figures” failed. The team can inspect that claim, determine whether the criterion or judgment is wrong, and revise the affected item instead of rewriting the entire evaluation. section 10.2.3 section 10.2.2
Aggregate disagreement patterns support targeted iteration: teams can revise an ambiguous checklist item, calculate failure rates by criterion, and detect regressions that an overall score would hide. The judge remains evidence, not truth. section 9.7.6 section 10.2.3
