liveBook · Manning

Automatic metric

As explained in Evaluating AI Systems

A metric computed directly from data by predefined formulas or algorithms without evaluative judgment during computation.

An automatic metric computes a score directly from data using a predefined formula or algorithm, without evaluative judgment during computation. This contrasts with judgment-based metrics, which require interpretation of qualities such as correctness, safety, or appropriateness. section 3.2

Lexical overlap for a medical chatbot

BLEU can compare a chatbot’s answers to reference answers from trusted medical sources or clinicians by measuring overlapping n-grams. It provides a lexical overlap signal; that signal alone does not establish whether a patient might act on dangerous advice. section 3.2 section 3.1

Automatic metrics are fast, inexpensive, reproducible, and easy to scale. They can support continuous monitoring and regression tracking, but may measure a proxy rather than the property that matters, especially for open-ended or subjective tasks. Before adopting one, inspect its definition and assumptions, and ask what its score means for users and risks. section 3.2 section 3.1

Table 3.2 Strengths, limitations, and typical use cases of common metric categories.
Metric category
Strengths
Limitations
When to consider
Automatic metrics
Fast, inexpensive, reproducible, easy to scale and automate
May capture proxy rather than desired properties, particularly for open-ended or subjective tasks
When you need fast, repeatable signals and when the property being evaluated can be measured objectively and reliably.
Judgment-Based metrics
Can capture contextual and subjective qualities and reveal failure modes that automatic metrics may overlook
More expensive, slower to collect, harder to scale, potentially sensitive to judge bias and variability
When evaluation outcomes carry high stakes, failures are costly or irreversible, or the quality of interest is inherently contextual or normative, such as safety, correctness, or user trust.
Reference-Based metrics
Easy to interpret, compare, benchmark, and track over time
Depends on the quality, completeness, and relevance of reference data; may penalize valid alternative outputs
When the task has a clear notion of correctness, high-quality references are available, and consistency and comparability across systems or time are important.
Reference-free metrics
Flexible, applicable when no ground truth exists, well suited for production monitoring and open-ended generation
Harder to validate, more dependent on assumptions about quality, often less comparable across systems
When evaluating open-ended generation, interactive behavior, or qualities such as safety, helpfulness, and user experience, especially when no single correct answer exists.

Compare automatic and judgment-based metrics by strengths, limitations, and suitable uses. section 3.2

How Evaluating AI Systems covers it

How deep it goes
Defines mechanical computation and contrasts it with evaluative judgment, without detailing metric implementation.
Examples
Uses BLEU n-gram overlap against reference answers for a medical advice chatbot.
What you can do after
Evaluate and understand when an automatic metric offers a useful signal and when its assumptions or proxy may mislead.
How it connects
Connects automatic metrics to judgment-based metrics, AI-as-a-judge, and retrieval-augmented generation faithfulness.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.