liveBook · Manning

Cohen's kappa

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Cohen's kappa is a statistic that measures agreement between two categorical annotators while correcting for agreement expected by chance.

Cohen's kappa (κ) measures categorical agreement between two evaluators while correcting for agreement expected from their marginal label frequencies. It compares observed agreement (Pₒ) with chance agreement (Pₑ): κ is positive when agreement exceeds chance, zero at chance, and negative below chance. section 4.1.4

Why percentage agreement can mislead

If 90% of outputs are acceptable and both evaluators randomly choose “acceptable” 90% of the time, they agree about 82% of the time by chance alone (0.9 × 0.9 + 0.1 × 0.1). Kappa discounts this baseline instead of treating the full percentage as meaningful agreement. section 4.1.4

Use kappa when the evaluation depends on categorical decisions:

  • Compare whether two evaluators assign the same labels for acceptability, hallucination, safety, or harm. section 4.1.4
  • Interpret κ in context rather than applying thresholds mechanically: 0.61–0.80 is commonly called substantial agreement, while task subjectivity changes what a score means. section 4.1.4
  • Treat kappa as evaluator consistency, not accuracy against ground truth; two evaluators can agree perfectly while being systematically wrong. section 4.1.4
  • Report confidence intervals, focus on effect size, and investigate disagreements alongside sample size and uncertainty. section 4.1.5
Table 4.4 Interpreting Cohen's Kappa values
Kappa (κ) Range
Interpretation
0.81 – 1.00
Almost perfect agreement
0.61 – 0.80
Substantial agreement
0.41 – 0.60
Moderate agreement
0.21 – 0.40
Fair agreement
0.00 – 0.20
Slight agreement
< 0.00
Disagreement is worse than chance

Use the interpretation table as a quick reference, then qualify the range with task subjectivity and decision consequences. section 4.1.4

Listing 4.3 Computing Cohen’s kappa with confidence intervals
import numpy as np
from scipy.stats import cohen_kappa_score
 
rater1 = [1, 2, 1, 2, 1, 2, 1, 2, 1, 2]              #A
rater2 = [1, 1, 2, 2, 1, 1, 2, 2, 1, 2]
 
kappa = cohen_kappa_score(rater1, rater2)
 
n_bootstraps = 1000
kappa_samples = []
for _ in range(n_bootstraps):
    indices = np.random.choice(len(rater1), size=len(rater1), replace=True)
    resampled_rater1 = [rater1[i] for i in indices]
    resampled_rater2 = [rater2[i] for i in indices]
    boot_kappa = cohen_kappa_score(resampled_rater1, resampled_rater2)
    kappa_samples.append(boot_kappa)
 
lower = np.percentile(kappa_samples, 2.5)
upper = np.percentile(kappa_samples, 97.5)
print(f"Cohen's kappa: {corr:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")

Compute kappa with SciPy and estimate a 95% confidence interval by bootstrapping paired evaluator labels. section 4.1.4 section 4.1.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines chance-corrected agreement, gives interpretation ranges and caveats, and demonstrates computation with bootstrap intervals.
Examples
Uses acceptability, hallucination, safety, harmfulness, and imbalanced 90%-acceptable labels as categorical evaluation examples.
What it assumes
Assumes categorical evaluator labels and introduces observed agreement, chance agreement, marginal frequencies, and ground-truth distinctions.
What you can do after
Diagnose evaluator consistency with Cohen's kappa, interpret disagreements, and compute bootstrap confidence intervals in Python.
How it connects
Connects categorical agreement with percentage agreement, ground truth, Fleiss' kappa, Spearman's rho, Kendall's tau, and confidence intervals.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.