liveBook · Manning

Metric

As explained in Evaluating AI Systems

A quantitative or scored measure used to assess a system against an evaluation criterion.

A metric is a quantitative or qualitative measure of how well a system performs on a specific task and evaluation dimension. It turns an abstract goal into observable evidence, but captures only part of system behavior and rests on assumptions about what matters and how to measure it. section 3.0 section 3.2

An evaluation dimension is what you want to assess; a metric is how you measure it. Safety, for example, is a dimension that can be measured through the rate of unsafe responses or the frequency of policy violations. Some terms serve both roles: accuracy can mean the broader goal of correctness or the proportion of correct predictions. section 2.5

A metric has four elements:

  • Input: the signals needed for measurement, such as system outputs, reference labels, judgments, or user interactions. section 3.2
  • Computation: the formula, algorithm, or judgment workflow that transforms the inputs. section 3.2
  • Output: a numerical or categorical result, such as a score, class-specific values, or a qualitative rating. section 3.2
  • Interpretation: what the result means in context, including its limitations and implications for decisions. section 3.2

Precision and faithfulness

For disease classification, precision divides true positive predictions by all positive predictions, indicating how often positive predictions correspond to actual cases. For a retrieval-augmented generation system, faithfulness assesses whether an answer is supported by retrieved context; a high score does not necessarily mean the answer is complete or useful. section 3.2

Automatic metrics can be fast, reproducible, and scalable, but may measure a proxy rather than the desired property. Judgment-based metrics can capture contextual qualities and overlooked failures, but are slower, more expensive, and potentially sensitive to judge bias or variability. The choice depends on the property, stakes, and resources. section 3.2 section 3.1

Select complementary metrics that support a specific decision and capture different successes and failure modes. A legal chatbot evaluated only for factual accuracy, for example, may still confuse users, mishandle ambiguity, or fail to refer them to a lawyer. Evaluate the system users encounter, not just its underlying model. section 3.0 section 1.5

Assess metric quality explicitly, including construct validity, reliability, robustness, interpretability, and measurability. When existing metrics miss an important requirement, define the target construct, choose an output format, specify the computation, pilot the metric, and document its assumptions and intended use. section 3.7

How Evaluating AI Systems covers it

How deep it goes
Defines metric components, contrasts measurement categories, and outlines quality criteria and custom metric design.
Examples
Uses disease classification precision, retrieval-augmented generation faithfulness, and medical chatbot measurement trade-offs.
What you can do after
Evaluate and select complementary metrics for a decision, and outline a custom metric when existing measures miss requirements.
How it connects
Connects metrics to evaluation dimensions, fairness, robustness, and human judgment.

Also explained in

Compare how 20 books explain metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Evaluating AI Systems. The section links point to the parts of the book it draws on.