When evaluators judge the same response differently, is the response ambiguous, or are the instructions unclear? Inter-rater reliability, also called inter-annotator agreement, measures how consistently annotators assign labels or judgments to the same items. Annotators can be human or automated. section 1.1.2
Agreement helps assess both the stability of a definition of “correct” and the quality of the resulting labels. Disagreement is not always an error: experts can interpret an inherently ambiguous signal differently. In one reported agreement scale, 0 represents random agreement, 1 perfect agreement, and 0.41–0.60 moderate agreement; interpret a score in the context of the measure used. section 1.1.2
To improve and use agreement in an evaluation workflow:
- Define the quality dimension concretely, specify observable criteria for each score, and provide examples and edge-case guidance. section 4.4.3
- Have multiple calibrated labelers judge the same items, then measure their agreement. Inspect disagreements for unclear instructions or genuinely ambiguous cases. section 1.1.2 section 8.3.2
- Monitor agreement as the labeler pool and rubric change. Drift can make it hard to tell whether a quality regression comes from the model, the rubric, the labelers, or a combination. section 7.4.3 section 8.1.1
Calibrating an AI judge
Sample 500–1,000 prompts from the production distribution and two responses per prompt. Have two or three calibrated human labelers rank the pairs, then compute human agreement. Run the AI labeler on the same pairs and compare its agreement with humans against the human-to-human rate. If it is materially lower, investigate vague principles, judge capability, or missing guidance for parts of the prompt distribution. section 8.3.2
Agreement in preference labeling
In the reported InstructGPT ranking tasks, inter-labeler agreement was roughly 73%: sufficient to train a reward model, but also a reminder that aggregated human preferences are noisy. Regular calibration and clear rubrics matter when scaling the labeler pool. section 7.5.2 section 7.4.3





