liveBook · Manning

Spearman's rank correlation coefficient

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Spearman's rank correlation coefficient measures the strength and direction of a monotonic relationship between two variables by correlating their rank positions.

Spearman's rank correlation coefficient (ρ) measures whether two evaluators order the same items consistently. It converts raw scores to ranks, then compares the rank differences. The measure ignores scoring-scale differences and exposes structural disagreements in ordering. section 4.1.2

Interpret ρ on a scale from −1 to 1:

  • ρ = 1 means identical rankings. section 4.1.2
  • ρ = −1 means perfectly reversed rankings. section 4.1.2
  • ρ = 0 means no systematic relationship between rankings. section 4.1.2
  • Large rank displacements matter more because the calculation squares rank differences. section 4.1.3

Human and LLM judge rankings

For five chatbot responses, human ranks are [1, 2, 5, 3, 4] and LLM-judge ranks are [1, 3, 5, 2, 4]. The squared rank differences sum to 2, producing ρ = 0.9. Both evaluators identify the same best and worst responses, but reverse the middle ordering of B and D. section 4.1.2

Table 4.1 Human expert vs. LLM judge ratings for five chatbot responses
Response
Description
Human
LLM Judge
A
Empathetic, resolves this issue
5
4.5
B
Correct but robotic
4
3.2
C
Completely wrong answer
1
1.5
D
Polite, but doesn’t solve the problem
3
4.0
E
Terse but accurate
2
2.0

Compare the underlying human and LLM-judge scores before converting them into ranks. section 4.1.1

For implementation, use scipy.stats.spearmanr to obtain ρ and a p-value, then bootstrap paired observations to estimate a 95% confidence interval. Report uncertainty and effect size together: small samples can produce unstable estimates, while statistical significance does not establish useful agreement. section 4.1.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains rank conversion, the squared-difference calculation, interpretation, implementation, bootstrapping, and sample-size caveats.
Examples
Uses five chatbot responses, human and LLM-judge ratings, rank differences, and a scipy bootstrap implementation.
What it assumes
Assumes paired evaluator scores and basic ranking, while introducing rank conversion, correlation interpretation, and confidence intervals.
What you can do after
Diagnose ranking agreement with scipy, quantify uncertainty through bootstrapping, and investigate disagreements rather than trusting one score.
How it connects
Contrasts with Kendall's tau and Cohen's kappa while connecting ranking agreement to LLM-as-a-judge evaluation and RLHF.

Also explained in

Compare how 3 books explain Spearman's rank correlation coefficient →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.