liveBook · Manning

Disagreement review

As explained in AI Model Evaluation

Disagreement analysis is the investigation of differing evaluation judgments to identify the specific output behavior or criterion responsible for the disagreement.

Disagreement review investigates cases where an LLM judge and human labels, gold examples, or another judge reach different conclusions. It treats each mismatch as a debugging signal rather than automatically trusting a fluent explanation or score. section 9.7.6 section 9.8

Review each disagreement by asking:

  • Did the judge apply the rubric correctly? section 9.7.6
  • Did it use the right evidence, or reward irrelevant preferences such as verbosity? section 9.7.6 section 9.8
  • Did missing context or an ambiguous rubric cause the mismatch? section 9.8
  • Can the failure be localized to one checklist criterion rather than an overall score? section 10.2.3

Localizing a summary disagreement

A human reviewer marks a summary accurate, but the judge fails it. A checklist may reveal that only “no unsupported figures” failed. The team can inspect that claim, determine whether the criterion or judgment is wrong, and revise the affected item instead of rewriting the entire evaluation. section 10.2.3 section 10.2.2

Aggregate disagreement patterns support targeted iteration: teams can revise an ambiguous checklist item, calculate failure rates by criterion, and detect regressions that an overall score would hide. The judge remains evidence, not truth. section 9.7.6 section 10.2.3

How AI Model Evaluation covers it

How deep it goes
Explains inspection questions, checklist localization, aggregate failure patterns, and regression detection; stops short of formal causal attribution.
Examples
Uses factual errors, verbosity bias, missing context, unsupported figures, incorrect attribution, and missing disclosures as disagreement cases.
What it assumes
Assumes human labels or gold examples, an evaluation rubric, and observable judge outputs; emphasizes validation before scale.
What you can do after
Diagnose judge failures, localize disputed output behavior, revise criteria, and monitor criterion-level regressions.
How it connects
Connects LLM-as-a-judge evaluation with human labels, rubrics, checklists, validation sets, and regression testing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.