Spearman's rank correlation coefficient (ρ) measures whether two evaluators order the same items consistently. It converts raw scores to ranks, then compares the rank differences. The measure ignores scoring-scale differences and exposes structural disagreements in ordering. section 4.1.2
Interpret ρ on a scale from −1 to 1:
- ρ = 1 means identical rankings. section 4.1.2
- ρ = −1 means perfectly reversed rankings. section 4.1.2
- ρ = 0 means no systematic relationship between rankings. section 4.1.2
- Large rank displacements matter more because the calculation squares rank differences. section 4.1.3
Human and LLM judge rankings
For five chatbot responses, human ranks are [1, 2, 5, 3, 4] and LLM-judge ranks are [1, 3, 5, 2, 4]. The squared rank differences sum to 2, producing ρ = 0.9. Both evaluators identify the same best and worst responses, but reverse the middle ordering of B and D. section 4.1.2
Table 4.1 Human expert vs. LLM judge ratings for five chatbot responses
|
Response
|
Description
|
Human
|
LLM Judge
|
|
A
|
Empathetic, resolves this issue
|
5
|
4.5
|
|
B
|
Correct but robotic
|
4
|
3.2
|
|
C
|
Completely wrong answer
|
1
|
1.5
|
|
D
|
Polite, but doesn’t solve the problem
|
3
|
4.0
|
|
E
|
Terse but accurate
|
2
|
2.0
|
Compare the underlying human and LLM-judge scores before converting them into ranks. section 4.1.1
For implementation, use scipy.stats.spearmanr to obtain ρ and a p-value, then bootstrap paired observations to estimate a 95% confidence interval. Report uncertainty and effect size together: small samples can produce unstable estimates, while statistical significance does not establish useful agreement. section 4.1.5


