liveBook · Manning

Inter-rater reliability

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Inter-annotator agreement is the degree to which independent annotators assign the same labels or judgments to the same items.

When evaluators judge the same response differently, is the response ambiguous, or are the instructions unclear? Inter-rater reliability, also called inter-annotator agreement, measures how consistently annotators assign labels or judgments to the same items. Annotators can be human or automated. section 1.1.2

Agreement helps assess both the stability of a definition of “correct” and the quality of the resulting labels. Disagreement is not always an error: experts can interpret an inherently ambiguous signal differently. In one reported agreement scale, 0 represents random agreement, 1 perfect agreement, and 0.41–0.60 moderate agreement; interpret a score in the context of the measure used. section 1.1.2

To improve and use agreement in an evaluation workflow:

  • Define the quality dimension concretely, specify observable criteria for each score, and provide examples and edge-case guidance. section 4.4.3
  • Have multiple calibrated labelers judge the same items, then measure their agreement. Inspect disagreements for unclear instructions or genuinely ambiguous cases. section 1.1.2 section 8.3.2
  • Monitor agreement as the labeler pool and rubric change. Drift can make it hard to tell whether a quality regression comes from the model, the rubric, the labelers, or a combination. section 7.4.3 section 8.1.1

Calibrating an AI judge

Sample 500–1,000 prompts from the production distribution and two responses per prompt. Have two or three calibrated human labelers rank the pairs, then compute human agreement. Run the AI labeler on the same pairs and compare its agreement with humans against the human-to-human rate. If it is materially lower, investigate vague principles, judge capability, or missing guidance for parts of the prompt distribution. section 8.3.2

Agreement in preference labeling

In the reported InstructGPT ranking tasks, inter-labeler agreement was roughly 73%: sufficient to train a reward model, but also a reminder that aggregated human preferences are noisy. Regular calibration and clear rubrics matter when scaling the labeler pool. section 7.5.2 section 7.4.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines an agreement scale and gives a calibration procedure, but does not specify a calculation formula.
Examples
Uses radiologist disagreement, InstructGPT preference rankings, and human-versus-AI judge comparisons as examples.
What you can do after
Diagnose/debug inconsistent judgments by examining rubric clarity, labeler calibration, and changes in the labeler pool.
How it connects
Connects agreement to evaluation rubrics, reward models, reinforcement learning from human feedback, and AI judges.

Also explained in

Compare how 6 books explain inter-rater reliability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.