A judgment-based metric assesses system behavior through interpretation, discretion, or normative reasoning. It can address whether an answer is correct, helpful, safe, or appropriate in context. section 3.2
Judges may use rubrics, rating scales, or comparisons. They can be people or AI systems instructed to approximate human judgment. Unlike automatic metrics computed mechanically from data, judgment-based metrics require interpretation. section 3.2
Evaluating a medical advice chatbot
Judges can assess whether an answer is supported by retrieved evidence and free of clinically unsafe claims. By contrast, BLEU can mechanically measure lexical overlap with trusted reference answers; overlap alone addresses a different property. section 3.2
Table 3.2 Strengths, limitations, and typical use cases of common metric categories.
|
Metric category
|
Strengths
|
Limitations
|
When to consider
|
|
Automatic metrics
|
Fast, inexpensive, reproducible, easy to scale and automate
|
May capture proxy rather than desired properties, particularly for open-ended or subjective tasks
|
When you need fast, repeatable signals and when the property being evaluated can be measured objectively and reliably.
|
|
Judgment-Based metrics
|
Can capture contextual and subjective qualities and reveal failure modes that automatic metrics may overlook
|
More expensive, slower to collect, harder to scale, potentially sensitive to judge bias and variability
|
When evaluation outcomes carry high stakes, failures are costly or irreversible, or the quality of interest is inherently contextual or normative, such as safety, correctness, or user trust.
|
|
Reference-Based metrics
|
Easy to interpret, compare, benchmark, and track over time
|
Depends on the quality, completeness, and relevance of reference data; may penalize valid alternative outputs
|
When the task has a clear notion of correctness, high-quality references are available, and consistency and comparability across systems or time are important.
|
|
Reference-free metrics
|
Flexible, applicable when no ground truth exists, well suited for production monitoring and open-ended generation
|
Harder to validate, more dependent on assumptions about quality, often less comparable across systems
|
When evaluating open-ended generation, interactive behavior, or qualities such as safety, helpfulness, and user experience, especially when no single correct answer exists.
|
Compare judgment-based metrics with other metric categories, including their strengths, limitations, and typical uses. section 3.2
Use judgment when the quality of interest is contextual or normative, or when failures carry high stakes. Account for slower collection, scaling costs, judge bias, and variability. section 3.2
