liveBook · Manning

Kendall's tau

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Kendall's rank correlation coefficient measures ordinal association by comparing concordant and discordant pairs of ranked observations.

Kendall's tau (τ) measures ranking agreement between two evaluators by comparing concordant pairs, ordered the same way, with discordant pairs, ordered differently. It ranges from −1, perfect opposite ordering, through 0, no relationship, to +1, perfect agreement. section 4.1.3

Use Kendall's tau to evaluate whether an automated evaluator preserves human preferences:

  • List every possible pair of ranked observations. section 4.1.3
  • Mark each pair concordant when both evaluators prefer the same item, or discordant when their preferences conflict. section 4.1.3
  • Compute τ as (concordant pairs − discordant pairs) divided by the total number of pairs. section 4.1.3
  • Interpret the result alongside sample size, confidence intervals, and other agreement measures rather than treating it as a universal quality score. section 4.1 section 4.1.3

Five chatbot responses

Five human-versus-LLM ratings produce 10 possible pairs. Nine pairs are concordant and one is discordant, giving τ = (9 − 1) / 10 = 0.8. Mapping τ from [−1, 1] to [0, 1] gives 0.9, an easier stakeholder-facing summary of pairwise agreement. section 4.1.3

Table 4.3 Concordant and discordant pairs from our chatbot example
Pair
Human says…
LLM-as-a-Judge says…
Verdict
A vs B
A better (rank 1 < 2)
A better (rank 1 < 3)
✅ Concordant
A vs C
A better (rank 1 < 5)
A better (rank 1 < 5)
✅ Concordant
A vs D
A better (rank 1 < 3)
A better (rank 1 < 2)
✅ Concordant
A vs E
A better (rank 1 < 4)
A better (rank 1 < 4)
✅ Concordant
B vs C
B better (rank 2 < 5)
B better (rank 3 < 5)
✅ Concordant
B vs D
B better (rank 2 < 3)
D better (rank 3 > 2)
❎ Discordant
B vs E
B better (rank 2 < 4)
B better (rank 3 < 4)
✅ Concordant
C vs D
D better (rank 5 > 3)
D better (rank 5 > 2)
✅ Concordant
C vs E
E better (rank 5 > 4)
E better (rank 5 > 4)
✅ Concordant
D vs E
D better (rank 3 > 4)
D better (rank 2 < 4)
✅ Concordant

Inspect the ten chatbot response pairs to see exactly how concordant and discordant judgments produce the score. section 4.1.3

Listing 4.2 Computing Kendall's tau with confidence intervals
import numpy as np
from scipy.stats import kendalltau
 
human_ranks = [1, 2, 3, 4, 5]
metric_ranks = [1, 3, 2, 5, 4]
 
tau, p_value = kendalltau(human_ranks, metric_ranks) 
 
n_bootstraps = 1000
tau_samples = []
for _ in range(n_bootstraps):
    indices = np.random.choice(len(human_ranks), len(human_ranks), replace=True)
    resampled_human = [human_ranks[i] for i in indices]
    resampled_metric = [metric_ranks[i] for i in indices]
    boot_tau, _ = kendalltau(resampled_human, resampled_metric)
    tau_samples.append(boot_tau)
 
lower = np.percentile(tau_samples, 2.5)
upper = np.percentile(tau_samples, 97.5)
print(f"Kendall's τ: {corr:.3f}, p-value: {p_value:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")

Run the Python implementation with SciPy's kendalltau and bootstrap resampling to obtain a tau estimate, p-value, and 95% confidence interval. section 4.1.3

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains pair counting, the formula, score interpretation, normalization, and bootstrap confidence intervals through a worked chatbot example.
Examples
Uses five chatbot responses, ten evaluator-comparison pairs, and a Python bootstrap implementation.
What it assumes
Assumes two evaluators produce comparable rankings and distinguishes ranking agreement from categorical label agreement.
What you can do after
Diagnose evaluator alignment with SciPy's kendalltau and quantify uncertainty through bootstrap confidence intervals.
How it connects
Connects ranking agreement with Spearman's rho, categorical agreement with Cohen's kappa, and preference optimization in RLHF.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.