Correlation measures how consistently two evaluators rank the same outputs, even when their scores use different absolute scales. In evaluation, it tests whether an automated metric orders responses like human judgment. section 4.1.1
Use correlation to compare evaluator rankings before interpreting score magnitudes:
- Spearman’s ρ measures whether two evaluators produce consistent rankings. section 4.1
- Kendall’s τ provides another rank-based comparison between evaluators. section 4.1
- Correlation differs from Cohen’s kappa, which measures agreement on categorical quality labels. section 4.1
Comparing a human evaluator with an LLM judge
Five chatbot responses receive human scores of 5, 4, 1, 3, and 2, while an LLM judge assigns 4.5, 3.2, 1.5, 4.0, and 2.0. The scores differ, but correlation asks whether both evaluators order the responses similarly. A strong correlation suggests alignment with human rankings; it does not prove that either evaluator is consistently correct. section 4.1.1
Table 4.1 Human expert vs. LLM judge ratings for five chatbot responses
|
Response
|
Description
|
Human
|
LLM Judge
|
|
A
|
Empathetic, resolves this issue
|
5
|
4.5
|
|
B
|
Correct but robotic
|
4
|
3.2
|
|
C
|
Completely wrong answer
|
1
|
1.5
|
|
D
|
Polite, but doesn’t solve the problem
|
3
|
4.0
|
|
E
|
Terse but accurate
|
2
|
2.0
|
Compare the five response descriptions and ratings used to illustrate evaluator ranking agreement. section 4.1.1





















