liveBook · Manning

Agreement metric

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An evaluator agreement metric quantifies the degree to which an evaluator's judgments agree with human preferences or judgments.

An agreement metric quantifies how closely an evaluator’s judgments match human preferences or judgments. It turns evaluator quality into a measurable alignment question rather than assuming that a model’s score is trustworthy. section 4.0 section 4.5

Common agreement metrics include:

  • Spearman’s ρ and Kendall’s τ compare evaluator rankings with human rankings. section 4.5
  • Cohen’s κ evaluates agreement when judgments are categorical. section 4.5
  • Interpret the metric as evidence about alignment, not as a universal measure of response quality. section 4.0 section 4.5

Checking an LLM judge

For a customer-service chatbot, compare the judge’s helpfulness and accuracy assessments with human judgments across many responses. A strong agreement result supports scalable evaluation; disagreement signals a need for calibration, bias checks, or revised criteria. section 4.0 section 4.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines statistical comparison and names three metrics, while deferring detailed calculation and interpretation procedures.
Examples
Uses customer-service response evaluation, ranking correlations, categorical judgments, and golden-set verification.
What it assumes
Assumes human judgments or preferences are available as a comparison basis.
What you can do after
Evaluate/understand whether an automated evaluator agrees sufficiently with human judgments for scalable assessment.
How it connects
Connects evaluator agreement metrics with human preferences, LLM-as-a-judge, G-Eval, calibration, and evaluator bias.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.