An automatic metric computes a score directly from data using a predefined formula or algorithm, without evaluative judgment during computation. This contrasts with judgment-based metrics, which require interpretation of qualities such as correctness, safety, or appropriateness. section 3.2
Lexical overlap for a medical chatbot
BLEU can compare a chatbot’s answers to reference answers from trusted medical sources or clinicians by measuring overlapping n-grams. It provides a lexical overlap signal; that signal alone does not establish whether a patient might act on dangerous advice. section 3.2 section 3.1
Automatic metrics are fast, inexpensive, reproducible, and easy to scale. They can support continuous monitoring and regression tracking, but may measure a proxy rather than the property that matters, especially for open-ended or subjective tasks. Before adopting one, inspect its definition and assumptions, and ask what its score means for users and risks. section 3.2 section 3.1
Table 3.2 Strengths, limitations, and typical use cases of common metric categories.
|
Metric category
|
Strengths
|
Limitations
|
When to consider
|
|
Automatic metrics
|
Fast, inexpensive, reproducible, easy to scale and automate
|
May capture proxy rather than desired properties, particularly for open-ended or subjective tasks
|
When you need fast, repeatable signals and when the property being evaluated can be measured objectively and reliably.
|
|
Judgment-Based metrics
|
Can capture contextual and subjective qualities and reveal failure modes that automatic metrics may overlook
|
More expensive, slower to collect, harder to scale, potentially sensitive to judge bias and variability
|
When evaluation outcomes carry high stakes, failures are costly or irreversible, or the quality of interest is inherently contextual or normative, such as safety, correctness, or user trust.
|
|
Reference-Based metrics
|
Easy to interpret, compare, benchmark, and track over time
|
Depends on the quality, completeness, and relevance of reference data; may penalize valid alternative outputs
|
When the task has a clear notion of correctness, high-quality references are available, and consistency and comparability across systems or time are important.
|
|
Reference-free metrics
|
Flexible, applicable when no ground truth exists, well suited for production monitoring and open-ended generation
|
Harder to validate, more dependent on assumptions about quality, often less comparable across systems
|
When evaluating open-ended generation, interactive behavior, or qualities such as safety, helpfulness, and user experience, especially when no single correct answer exists.
|
Compare automatic and judgment-based metrics by strengths, limitations, and suitable uses. section 3.2
