liveBook · Manning

Judgment-based metric

As explained in Evaluating AI Systems

A metric whose assessment depends on evaluative judgments involving interpretation, discretion, or normative reasoning.

A judgment-based metric assesses system behavior through interpretation, discretion, or normative reasoning. It can address whether an answer is correct, helpful, safe, or appropriate in context. section 3.2

Judges may use rubrics, rating scales, or comparisons. They can be people or AI systems instructed to approximate human judgment. Unlike automatic metrics computed mechanically from data, judgment-based metrics require interpretation. section 3.2

Evaluating a medical advice chatbot

Judges can assess whether an answer is supported by retrieved evidence and free of clinically unsafe claims. By contrast, BLEU can mechanically measure lexical overlap with trusted reference answers; overlap alone addresses a different property. section 3.2

Table 3.2 Strengths, limitations, and typical use cases of common metric categories.
Metric category
Strengths
Limitations
When to consider
Automatic metrics
Fast, inexpensive, reproducible, easy to scale and automate
May capture proxy rather than desired properties, particularly for open-ended or subjective tasks
When you need fast, repeatable signals and when the property being evaluated can be measured objectively and reliably.
Judgment-Based metrics
Can capture contextual and subjective qualities and reveal failure modes that automatic metrics may overlook
More expensive, slower to collect, harder to scale, potentially sensitive to judge bias and variability
When evaluation outcomes carry high stakes, failures are costly or irreversible, or the quality of interest is inherently contextual or normative, such as safety, correctness, or user trust.
Reference-Based metrics
Easy to interpret, compare, benchmark, and track over time
Depends on the quality, completeness, and relevance of reference data; may penalize valid alternative outputs
When the task has a clear notion of correctness, high-quality references are available, and consistency and comparability across systems or time are important.
Reference-free metrics
Flexible, applicable when no ground truth exists, well suited for production monitoring and open-ended generation
Harder to validate, more dependent on assumptions about quality, often less comparable across systems
When evaluating open-ended generation, interactive behavior, or qualities such as safety, helpfulness, and user experience, especially when no single correct answer exists.

Compare judgment-based metrics with other metric categories, including their strengths, limitations, and typical uses. section 3.2

Use judgment when the quality of interest is contextual or normative, or when failures carry high stakes. Account for slower collection, scaling costs, judge bias, and variability. section 3.2

How Evaluating AI Systems covers it

How deep it goes
Defines judgment-based metrics, contrasts automatic computation, and describes judging methods and practical limitations.
Examples
Illustrates judging a medical chatbot's faithfulness and safety, alongside BLEU comparison with reference answers.
What you can do after
Evaluate and understand when judgment-based metrics are useful and what costs, bias, and variability to consider.
How it connects
Connects judgment-based metrics to AI-as-a-judge, RAG faithfulness, automatic metrics, and reference-based metrics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.