liveBook · Manning

Correlation

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A correlation coefficient is a numerical measure of the consistency of the relationship or ordering between two sets of scores, independent of their absolute scales.

Correlation measures how consistently two evaluators rank the same outputs, even when their scores use different absolute scales. In evaluation, it tests whether an automated metric orders responses like human judgment. section 4.1.1

Use correlation to compare evaluator rankings before interpreting score magnitudes:

  • Spearman’s ρ measures whether two evaluators produce consistent rankings. section 4.1
  • Kendall’s τ provides another rank-based comparison between evaluators. section 4.1
  • Correlation differs from Cohen’s kappa, which measures agreement on categorical quality labels. section 4.1

Comparing a human evaluator with an LLM judge

Five chatbot responses receive human scores of 5, 4, 1, 3, and 2, while an LLM judge assigns 4.5, 3.2, 1.5, 4.0, and 2.0. The scores differ, but correlation asks whether both evaluators order the responses similarly. A strong correlation suggests alignment with human rankings; it does not prove that either evaluator is consistently correct. section 4.1.1

Table 4.1 Human expert vs. LLM judge ratings for five chatbot responses
Response
Description
Human
LLM Judge
A
Empathetic, resolves this issue
5
4.5
B
Correct but robotic
4
3.2
C
Completely wrong answer
1
1.5
D
Polite, but doesn’t solve the problem
3
4.0
E
Terse but accurate
2
2.0

Compare the five response descriptions and ratings used to illustrate evaluator ranking agreement. section 4.1.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines rank consistency, distinguishes it from categorical agreement, and works through a five-response example.
Examples
Uses customer-service chatbot ratings from a human expert and an LLM judge across five responses.
What it assumes
Assumes paired evaluator scores and introduces rank comparison before discussing statistical interpretation.
What you can do after
Evaluate whether an automated metric orders outputs consistently with human judgments.
How it connects
Connects correlation with Spearman’s rho, Kendall’s tau, human evaluation, LLM-as-a-judge, ranking, and reward modeling.

Also explained in

Compare how 22 books explain correlation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.