Cohen's kappa (κ) measures categorical agreement between two evaluators while correcting for agreement expected from their marginal label frequencies. It compares observed agreement (Pₒ) with chance agreement (Pₑ): κ is positive when agreement exceeds chance, zero at chance, and negative below chance. section 4.1.4
Why percentage agreement can mislead
If 90% of outputs are acceptable and both evaluators randomly choose “acceptable” 90% of the time, they agree about 82% of the time by chance alone (0.9 × 0.9 + 0.1 × 0.1). Kappa discounts this baseline instead of treating the full percentage as meaningful agreement. section 4.1.4
Use kappa when the evaluation depends on categorical decisions:
- Compare whether two evaluators assign the same labels for acceptability, hallucination, safety, or harm. section 4.1.4
- Interpret κ in context rather than applying thresholds mechanically: 0.61–0.80 is commonly called substantial agreement, while task subjectivity changes what a score means. section 4.1.4
- Treat kappa as evaluator consistency, not accuracy against ground truth; two evaluators can agree perfectly while being systematically wrong. section 4.1.4
- Report confidence intervals, focus on effect size, and investigate disagreements alongside sample size and uncertainty. section 4.1.5
Table 4.4 Interpreting Cohen's Kappa values
|
Kappa (κ) Range
|
Interpretation
|
|
0.81 – 1.00
|
Almost perfect agreement
|
|
0.61 – 0.80
|
Substantial agreement
|
|
0.41 – 0.60
|
Moderate agreement
|
|
0.21 – 0.40
|
Fair agreement
|
|
0.00 – 0.20
|
Slight agreement
|
|
< 0.00
|
Disagreement is worse than chance
|
Use the interpretation table as a quick reference, then qualify the range with task subjectivity and decision consequences. section 4.1.4
Listing 4.3 Computing Cohen’s kappa with confidence intervals
import numpy as np
from scipy.stats import cohen_kappa_score
rater1 = [1, 2, 1, 2, 1, 2, 1, 2, 1, 2] #A
rater2 = [1, 1, 2, 2, 1, 1, 2, 2, 1, 2]
kappa = cohen_kappa_score(rater1, rater2)
n_bootstraps = 1000
kappa_samples = []
for _ in range(n_bootstraps):
indices = np.random.choice(len(rater1), size=len(rater1), replace=True)
resampled_rater1 = [rater1[i] for i in indices]
resampled_rater2 = [rater2[i] for i in indices]
boot_kappa = cohen_kappa_score(resampled_rater1, resampled_rater2)
kappa_samples.append(boot_kappa)
lower = np.percentile(kappa_samples, 2.5)
upper = np.percentile(kappa_samples, 97.5)
print(f"Cohen's kappa: {corr:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
Compute kappa with SciPy and estimate a 95% confidence interval by bootstrapping paired evaluator labels. section 4.1.4 section 4.1.5
