Inter-rater reliability
Inter-rater reliability is the degree to which independent raters give consistent judgments or labels to the same items, subjects, or events.
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
A measure of annotators’ agreement on labels assigned to tokens, considered separately from span boundaries.
- How it is used
- Measures annotation reliability and directs review of token-label and span-boundary disagreements.
- How deep it goes
- Explains chance-adjusted agreement and Krippendorff’s alpha, with separate token and span calculations for sequence labeling.
- Examples
- Illustrates token and span agreement using competing location annotations for “San” and “San Francisco.”
- What you can do after
- Evaluate annotation reliability and route disputed spans to adjudication without reannotating whole documents.
LLM Evaluation and Alignment, The Foundational Ideas
Inter-annotator agreement is the degree to which independent annotators assign the same labels or judgments to the same items.
- How it is used
- Assesses consistency of human or automated judgments in open-ended evaluation, preference labeling, and AI-judge calibration.
- How deep it goes
- Defines an agreement scale and gives a calibration procedure, but does not specify a calculation formula.
- Examples
- Uses radiologist disagreement, InstructGPT preference rankings, and human-versus-AI judge comparisons as examples.
- What you can do after
- Diagnose/debug inconsistent judgments by examining rubric clarity, labeler calibration, and changes in the labeler pool.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains inter-rater reliability →
Effective Data Analysis
Inter-rater reliability is the degree of agreement or consistency among independent raters evaluating the same subjects or events.
- How it is used
- Assess consistency between raters when evaluating qualitative data.
- How deep it goes
- Define agreement between raters and illustrate variation in topic labels, without specifying a calculation method.
- Examples
- Compare raters’ labels for service, product, and initial setup in qualitative data.
- What you can do after
- Evaluate how consistently raters assign judgments to qualitative data.
Read how Effective Data Analysis explains inter-rater reliability →
Machine Learning System Design
Interrater reliability is the degree to which multiple raters produce consistent or mutually agreeing judgments on the same items.
- How it is used
- Measures whether a labeling team produces consistent judgments on the same items.
- How deep it goes
- Names several reliability metrics and their uses, but does not explain their calculation.
- Examples
- Names Cohen’s kappa, Scott’s pi, and Krippendorff’s alpha as reliability metrics.
- What you can do after
- Evaluate labeling consistency, filter unreliable labels, and estimate a possible upper bound for model performance.
Read how Machine Learning System Design explains inter-rater reliability →
LLM Customization and Fine-Tuning
The degree to which different annotators assign the same labels or answers to the same examples.
- How it is used
- Checks whether human-labeled data is consistent enough to support model adaptation.
- How deep it goes
- Measures agreement with Cohen’s kappa, flags vague briefs below 0.60, and relates human agreement to a rough performance ceiling.
- Examples
- Uses ticket triage labels for network, access, hardware, and software issues, with severity levels P1 through P4.
- What you can do after
- Evaluate annotation consistency with Cohen’s kappa and improve it through clearer label definitions, boundary rules, and worked examples.
Read how LLM Customization and Fine-Tuning explains inter-rater reliability →
Building Specialized AI Systems
Inter-annotator agreement is a measure of how consistently different annotators assign labels to the same examples.
- How it is used
- Supports labeling quality control before supervised fine-tuning.
- How deep it goes
- Defines consistency checks and their diagnostic meaning, but does not specify a statistical calculation.
- Examples
- Uses repeated annotation, transaction classification, and compliance-regulation interpretation as quality-control examples.
- What you can do after
- Diagnose/debug with it by locating inconsistent annotations and improving labeling instructions.
Read how Building Specialized AI Systems explains inter-rater reliability →
Also covered in
What to know first
- Cohen's kappaCohen’s kappa is a statistic that measures agreement between two raters assigning categorical ratings, adjusted for the agreement expected by chance.
- Krippendorff's alphaKrippendorff’s alpha is a reliability statistic that measures agreement among raters while accounting for chance agreement and accommodating different measurement scales and missing data.
- labelerA labeler is a person or process that assigns labels, ratings, judgments, or other structured annotations to data or model outputs.
- biasBias is a systematic tendency or deviation that distorts measurements, estimates, judgments, or outcomes relative to a relevant reference or target.
Related concepts
- accuracycontrasts with · Accuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- Cohen's kappameasured by · Cohen’s kappa is a statistic that measures agreement between two raters assigning categorical ratings, adjusted for the agreement expected by chance.
- ground truthuses · Ground truth is reference information treated as correct for assessing the accuracy or correctness of predictions, labels, or other outputs.
- reliabilitykind of · Reliability is the degree to which a system, process, or measurement performs or yields consistent, dependable results under specified conditions.
- data annotationmeasures · Data annotation is the process of assigning labels, metadata, or structured judgments to data according to specified criteria.
Questions about inter-rater reliability
What is inter-rater reliability?
Inter-rater reliability is the degree to which independent raters give consistent judgments or labels to the same items, subjects, or events.
Which Manning books explain inter-rater reliability?
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI, LLM Evaluation and Alignment, The Foundational Ideas, Effective Data Analysis, Machine Learning System Design, LLM Customization and Fine-Tuning and Building Specialized AI Systems.
What should I know before learning inter-rater reliability?
Manning books that teach inter-rater reliability usually introduce Cohen's kappa, Krippendorff's alpha, labeler and bias first.
What is the difference between inter-rater reliability and accuracy?
Inter-rater reliability is the degree to which independent raters give consistent judgments or labels to the same items, subjects, or events. Accuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
What is inter-rater reliability a kind of?
Inter-rater reliability is a kind of reliability. Reliability is the degree to which a system, process, or measurement performs or yields consistent, dependable results under specified conditions.
What is inter-rater reliability also called?
Inter-rater reliability is also called inter-annotator agreement or inter-rater agreement.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





