liveBook · Manning

Inter-rater reliability

Also called inter-annotator agreement, inter-rater agreement

Inter-rater reliability is the degree to which independent raters give consistent judgments or labels to the same items, subjects, or events.

How 6 Manning books explain it

Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI

A measure of annotators’ agreement on labels assigned to tokens, considered separately from span boundaries.

How it is used
Measures annotation reliability and directs review of token-label and span-boundary disagreements.
How deep it goes
Explains chance-adjusted agreement and Krippendorff’s alpha, with separate token and span calculations for sequence labeling.
Examples
Illustrates token and span agreement using competing location annotations for “San” and “San Francisco.”
What you can do after
Evaluate annotation reliability and route disputed spans to adjudication without reannotating whole documents.

Read how Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI explains inter-rater reliability →

LLM Evaluation and Alignment, The Foundational Ideas

Inter-annotator agreement is the degree to which independent annotators assign the same labels or judgments to the same items.

How it is used
Assesses consistency of human or automated judgments in open-ended evaluation, preference labeling, and AI-judge calibration.
How deep it goes
Defines an agreement scale and gives a calibration procedure, but does not specify a calculation formula.
Examples
Uses radiologist disagreement, InstructGPT preference rankings, and human-versus-AI judge comparisons as examples.
What you can do after
Diagnose/debug inconsistent judgments by examining rubric clarity, labeler calibration, and changes in the labeler pool.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains inter-rater reliability →

Effective Data Analysis

Inter-rater reliability is the degree of agreement or consistency among independent raters evaluating the same subjects or events.

How it is used
Assess consistency between raters when evaluating qualitative data.
How deep it goes
Define agreement between raters and illustrate variation in topic labels, without specifying a calculation method.
Examples
Compare raters’ labels for service, product, and initial setup in qualitative data.
What you can do after
Evaluate how consistently raters assign judgments to qualitative data.

Read how Effective Data Analysis explains inter-rater reliability →

Machine Learning System Design

Interrater reliability is the degree to which multiple raters produce consistent or mutually agreeing judgments on the same items.

How it is used
Measures whether a labeling team produces consistent judgments on the same items.
How deep it goes
Names several reliability metrics and their uses, but does not explain their calculation.
Examples
Names Cohen’s kappa, Scott’s pi, and Krippendorff’s alpha as reliability metrics.
What you can do after
Evaluate labeling consistency, filter unreliable labels, and estimate a possible upper bound for model performance.

Read how Machine Learning System Design explains inter-rater reliability →

LLM Customization and Fine-Tuning

The degree to which different annotators assign the same labels or answers to the same examples.

How it is used
Checks whether human-labeled data is consistent enough to support model adaptation.
How deep it goes
Measures agreement with Cohen’s kappa, flags vague briefs below 0.60, and relates human agreement to a rough performance ceiling.
Examples
Uses ticket triage labels for network, access, hardware, and software issues, with severity levels P1 through P4.
What you can do after
Evaluate annotation consistency with Cohen’s kappa and improve it through clearer label definitions, boundary rules, and worked examples.

Read how LLM Customization and Fine-Tuning explains inter-rater reliability →

Building Specialized AI Systems

Inter-annotator agreement is a measure of how consistently different annotators assign labels to the same examples.

How it is used
Supports labeling quality control before supervised fine-tuning.
How deep it goes
Defines consistency checks and their diagnostic meaning, but does not specify a statistical calculation.
Examples
Uses repeated annotation, transaction classification, and compliance-regulation interpretation as quality-control examples.
What you can do after
Diagnose/debug with it by locating inconsistent annotations and improving labeling instructions.

Read how Building Specialized AI Systems explains inter-rater reliability →

Also covered in

What to know first

Related concepts

Questions about inter-rater reliability

What is inter-rater reliability?

Inter-rater reliability is the degree to which independent raters give consistent judgments or labels to the same items, subjects, or events.

Which Manning books explain inter-rater reliability?

Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI, LLM Evaluation and Alignment, The Foundational Ideas, Effective Data Analysis, Machine Learning System Design, LLM Customization and Fine-Tuning and Building Specialized AI Systems.

What should I know before learning inter-rater reliability?

Manning books that teach inter-rater reliability usually introduce Cohen's kappa, Krippendorff's alpha, labeler and bias first.

What is the difference between inter-rater reliability and accuracy?

Inter-rater reliability is the degree to which independent raters give consistent judgments or labels to the same items, subjects, or events. Accuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.

What is inter-rater reliability a kind of?

Inter-rater reliability is a kind of reliability. Reliability is the degree to which a system, process, or measurement performs or yields consistent, dependable results under specified conditions.

What is inter-rater reliability also called?

Inter-rater reliability is also called inter-annotator agreement or inter-rater agreement.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.