liveBook · Manning

Inter-rater reliability

As explained in Machine Learning System Design

Interrater reliability is the degree to which multiple raters produce consistent or mutually agreeing judgments on the same items.

Inter-rater reliability measures how consistently multiple labelers judge the same items. It provides a quality check before unreliable labels weaken an ML system. section 6.2.4

Use it to:

  • Measure labeler consistency and mutual agreement. section 6.2.4
  • Filter out unscrupulous or unreliable labels. section 6.2.4
  • Estimate a possible upper bound for model performance in some scenarios. section 6.2.4

Common metrics

Cohen’s kappa, Scott’s pi, and Krippendorff’s alpha are named measures of inter-rater reliability, distinct from regular ML metrics. section 6.2.4

How Machine Learning System Design covers it

How deep it goes
Introduces separate reliability metrics and their purposes without deriving formulas or prescribing a calculation procedure.
Examples
Names Cohen’s kappa, Scott’s pi, and Krippendorff’s alpha as representative measures.
What you can do after
Evaluate label consistency, filter unreliable annotations, and estimate human-agreement limits on model performance.
How it connects
Connects labeling with crowdsourcing, ML metrics, human-level performance, and model evaluation.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

Compare how 2 books explain inter-rater reliability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.