Kendall's tau (τ) measures ranking agreement between two evaluators by comparing concordant pairs, ordered the same way, with discordant pairs, ordered differently. It ranges from −1, perfect opposite ordering, through 0, no relationship, to +1, perfect agreement. section 4.1.3
Use Kendall's tau to evaluate whether an automated evaluator preserves human preferences:
- List every possible pair of ranked observations. section 4.1.3
- Mark each pair concordant when both evaluators prefer the same item, or discordant when their preferences conflict. section 4.1.3
- Compute τ as (concordant pairs − discordant pairs) divided by the total number of pairs. section 4.1.3
- Interpret the result alongside sample size, confidence intervals, and other agreement measures rather than treating it as a universal quality score. section 4.1 section 4.1.3
Five chatbot responses
Five human-versus-LLM ratings produce 10 possible pairs. Nine pairs are concordant and one is discordant, giving τ = (9 − 1) / 10 = 0.8. Mapping τ from [−1, 1] to [0, 1] gives 0.9, an easier stakeholder-facing summary of pairwise agreement. section 4.1.3
Table 4.3 Concordant and discordant pairs from our chatbot example
|
Pair
|
Human says…
|
LLM-as-a-Judge says…
|
Verdict
|
|
A vs B
|
A better (rank 1 < 2)
|
A better (rank 1 < 3)
|
✅
Concordant
|
|
A vs C
|
A better (rank 1 < 5)
|
A better (rank 1 < 5)
|
✅
Concordant
|
|
A vs D
|
A better (rank 1 < 3)
|
A better (rank 1 < 2)
|
✅
Concordant
|
|
A vs E
|
A better (rank 1 < 4)
|
A better (rank 1 < 4)
|
✅
Concordant
|
|
B vs C
|
B better (rank 2 < 5)
|
B better (rank 3 < 5)
|
✅
Concordant
|
|
B vs D
|
B better (rank 2 < 3)
|
D better (rank 3 > 2)
|
❎
Discordant
|
|
B vs E
|
B better (rank 2 < 4)
|
B better (rank 3 < 4)
|
✅
Concordant
|
|
C vs D
|
D better (rank 5 > 3)
|
D better (rank 5 > 2)
|
✅
Concordant
|
|
C vs E
|
E better (rank 5 > 4)
|
E better (rank 5 > 4)
|
✅
Concordant
|
|
D vs E
|
D better (rank 3 > 4)
|
D better (rank 2 < 4)
|
✅
Concordant
|
Inspect the ten chatbot response pairs to see exactly how concordant and discordant judgments produce the score. section 4.1.3
Listing 4.2 Computing Kendall's tau with confidence intervals
import numpy as np
from scipy.stats import kendalltau
human_ranks = [1, 2, 3, 4, 5]
metric_ranks = [1, 3, 2, 5, 4]
tau, p_value = kendalltau(human_ranks, metric_ranks)
n_bootstraps = 1000
tau_samples = []
for _ in range(n_bootstraps):
indices = np.random.choice(len(human_ranks), len(human_ranks), replace=True)
resampled_human = [human_ranks[i] for i in indices]
resampled_metric = [metric_ranks[i] for i in indices]
boot_tau, _ = kendalltau(resampled_human, resampled_metric)
tau_samples.append(boot_tau)
lower = np.percentile(tau_samples, 2.5)
upper = np.percentile(tau_samples, 97.5)
print(f"Kendall's τ: {corr:.3f}, p-value: {p_value:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
Run the Python implementation with SciPy's kendalltau and bootstrap resampling to obtain a tau estimate, p-value, and 95% confidence interval. section 4.1.3
