An agreement metric quantifies how closely an evaluator’s judgments match human preferences or judgments. It turns evaluator quality into a measurable alignment question rather than assuming that a model’s score is trustworthy. section 4.0 section 4.5
Common agreement metrics include:
- Spearman’s ρ and Kendall’s τ compare evaluator rankings with human rankings. section 4.5
- Cohen’s κ evaluates agreement when judgments are categorical. section 4.5
- Interpret the metric as evidence about alignment, not as a universal measure of response quality. section 4.0 section 4.5
Checking an LLM judge
For a customer-service chatbot, compare the judge’s helpfulness and accuracy assessments with human judgments across many responses. A strong agreement result supports scalable evaluation; disagreement signals a need for calibration, bias checks, or revised criteria. section 4.0 section 4.5
