Inter-rater reliability measures whether different annotators agree on labels for the same examples. Check agreement on sample tasks: when people disagree about the correct output, the model’s performance ceiling is limited by that disagreement. section 5.1.7 section 3.4.1
For ticket labeling, make agreement actionable:
- Measure agreement with Cohen’s kappa. A vague labeling brief can yield a kappa below 0.60, indicating meaningful disagreement. section 3.4.1
- Clarify label definitions and recurring boundary cases, then provide three to five worked examples for each label. section 3.4.1
- For a golden evaluation set, use fully human-annotated labels, even if training data was assembled with human validation. section 3.4.1





