liveBook · Manning

Inter-rater reliability

As explained in LLM Customization and Fine-Tuning

The degree to which different annotators assign the same labels or answers to the same examples.

Inter-rater reliability measures whether different annotators agree on labels for the same examples. Check agreement on sample tasks: when people disagree about the correct output, the model’s performance ceiling is limited by that disagreement. section 5.1.7 section 3.4.1

For ticket labeling, make agreement actionable:

  • Measure agreement with Cohen’s kappa. A vague labeling brief can yield a kappa below 0.60, indicating meaningful disagreement. section 3.4.1
  • Clarify label definitions and recurring boundary cases, then provide three to five worked examples for each label. section 3.4.1
  • For a golden evaluation set, use fully human-annotated labels, even if training data was assembled with human validation. section 3.4.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Measures agreement with Cohen’s kappa, flags vague briefs below 0.60, and relates human agreement to a rough performance ceiling.
Examples
Uses ticket triage labels for network, access, hardware, and software issues, with severity levels P1 through P4.
What it assumes
Requires multiple annotators to label sample tasks and recommends consistent labeling guidelines.
What you can do after
Evaluate annotation consistency with Cohen’s kappa and improve it through clearer label definitions, boundary rules, and worked examples.
How it connects
Links human annotation to Cohen’s kappa, labeling briefs, ground truth, and evaluation sets.

Also explained in

Compare how 6 books explain inter-rater reliability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.