Chapter 3: Picking evaluation metrics
The introduction explains how metrics turn evaluation goals into evidence and why choosing them requires more than picking a familiar measure.
After this chapter you will be able to
- Metrics as observable evidence. Explain that a metric quantifies an aspect of system behavior, making an abstract evaluation goal observable, and that different metrics capture different aspects of performance.
- Purposeful metric selection. Justify choosing a set of metrics around the evaluation decisions and context rather than defaulting to popular or convenient measures. Describe the need for the set to capture relevant successes and failures and be understandable to decision-makers.
- Automatic and human judgment tradeoffs. Explain why the team is divided between automatic metrics and human judgments when evaluating the medical advice chatbot.
- Contextual fit of metrics. Explain why the team cannot assume an off-the-shelf metric is suitable for the chatbot evaluation.
- Task and dimension measurement. Explain that a metric measures how well an AI system performs on a particular task and evaluation dimension, and that its result may be quantitative or qualitative.
- Four metric elements. Describe a metric’s inputs, computation method, output, and interpretation, including why interpreting a score requires attention to what it does not capture.
- Mechanical versus judgment scoring. Distinguish automatic metrics from judgment-based metrics, and describe how human or AI judges can supply the evaluative judgment.
- Reference use in metrics. Distinguish reference-based metrics from reference-free metrics and explain why the distinction matters when acceptable answers are unavailable or costly to produce.
- Execution-based outcome evaluation. Explain what execution-based metrics assess and balance their end-to-end advantages against their infrastructure costs and inability to establish safety on their own.
- Measurability versus metric quality. Explain why being measurable is not enough to make a metric useful for evaluating an AI system.
- Metric and construct alignment. Explain that construct validity is how well a metric captures its intended quality, and distinguish a metric that tracks that quality from a convenient but misleading proxy.
- Construct validity checks. Describe how to check a metric’s construct validity by tracing its connection to the desired outcome, reviewing misleading results in similar contexts, and comparing its scores with human or domain judgments.
- Meaning of causal proximity. Distinguish causal proximity from construct validity and explain why causal proximity matters when using a metric to improve a system.
- Tests for causal proximity. Describe how controlled quality improvements and irrelevant variations can test whether a metric responds for the right reasons.
- Feedback effect on precision. Trace how acting on churn predictions can lower measured precision even when retention succeeds.
- Meaning of metric movability. Explain that movability is a metric’s ability to register meaningful improvements, including small gains as a system matures.
- Checks for lost sensitivity. Describe how score distributions, controlled ablations, and trends across releases can reveal whether a metric still responds to meaningful performance changes.
- Metric reliability and reproducibility. Explain that a reliable metric gives similar scores when an evaluation is repeated under the same conditions, and why that consistency matters for interpreting results.
- Practical reliability checks. Describe how to assess reliability by repeating an evaluation with identical inputs and settings and, for judgment-based metrics, checking agreement across judges.
- Metric robustness and fragility. Explain that a robust metric remains trustworthy when small, irrelevant aspects of the system, data, or environment change, while a fragile metric shows misleading score swings. Describe why those swings make it harder to trust results and track real progress.
- Methods for testing robustness. Describe perturbation testing, cross-environment validation, and resampling as ways to check whether metric results hold under controlled minor changes, different evaluation settings, and different samples of data.
- Metric and human judgment alignment. Explain what it means for metric scores to correlate with human judgments and why poor alignment can make optimizing a metric counterproductive.
- Explainable metrics and their value. Explain why a metric must be understandable as well as accurate, especially when evaluation results affect high-stakes decisions.
- Metrics for different evaluation purposes. Distinguish the metrics to prioritize for stakeholder-facing evaluations from those useful for research or diagnosis, and describe how to communicate complex results.
- Practical metric measurability. Explain measurability as the ability to compute a metric easily and reliably within acceptable effort, cost, time, and resource limits. Describe why choosing a metric requires balancing rigor and completeness against practical feasibility.
- McNamara fallacy in evaluation. Explain how the body-count example illustrates the danger of favoring an easy-to-track metric over harder-to-measure objectives. Apply that warning to selecting AI metrics that reflect the system behavior relevant to a decision.
- Limits of off-the-shelf metrics. Explain when an existing metric may fail to fit an evaluation goal and why adapting it or creating a new metric may be appropriate.
- Observable components of constructs. Describe how to turn an abstract target construct into components that can be observed and assessed.
- Ongoing metric reassessment. Explain why a metric that once worked may need to be revalidated, revised, retired, or replaced as the system, its use, or the evaluation goal changes.
- Metric version traceability. Justify explicitly versioning evolving metrics and recording the version used for every evaluation result.
What it builds on
- Choosing evaluation dimensions (chapter 2) · The aspects of performance selected for evaluation.
- Defining evaluation goals (chapter 2) · The evaluation goals that determine which decisions need support.
- Defining the task to be evaluated (chapter 2) · The task to be evaluated.
- Deciding go/no-go (chapter 2) · the go/no-go evaluation goal
Section by section
Introduction
The introduction explains how metrics turn evaluation goals into evidence and why choosing them requires more than picking a familiar measure.
- Metrics as observable evidence: Explain that a metric quantifies an aspect of system behavior, making an abstract evaluation goal observable, and that different metrics capture different aspects of performance.
Why it matters: This helps you interpret what a metric does—and does not—tell you about a system. - Purposeful metric selection: Justify choosing a set of metrics around the evaluation decisions and context rather than defaulting to popular or convenient measures. Describe the need for the set to capture relevant successes and failures and be understandable to decision-makers.
Why it matters: A convenient metric is useful only if the resulting evidence supports the decision you actually need to make.
Key terms: metric, accuracy, task success rate, observable evidence, set of metrics, evaluation goals, successes and failures
3.1 “How do we measure this?”
A medical-chatbot evaluation meeting shows why choosing metrics requires balancing practical measurement with whether the scores reflect the properties that matter.
- Automatic and human judgment tradeoffs: Explain why the team is divided between automatic metrics and human judgments when evaluating the medical advice chatbot.
Why it matters: A metric needs to be practical to use without losing sight of what its result means for patients. - Contextual fit of metrics: Explain why the team cannot assume an off-the-shelf metric is suitable for the chatbot evaluation.
Why it matters: A convenient metric can lead the team astray if it does not reflect the concern they intend to assess.
Key terms: off-the-shelf automatic metrics, human judgments, judgment-based evaluations, off-the-shelf metrics, assumptions, properties we care about
3.2 What is a metric
This section defines an AI evaluation metric, explains its four elements, and distinguishes major metric categories and their limitations.
- Task and dimension measurement: Explain that a metric measures how well an AI system performs on a particular task and evaluation dimension, and that its result may be quantitative or qualitative.
Why it matters: A metric is meaningful only in relation to what the system is doing and what aspect of performance is being evaluated. - Four metric elements: Describe a metric’s inputs, computation method, output, and interpretation, including why interpreting a score requires attention to what it does not capture.
Why it matters: A score alone cannot tell you how it was produced or what conclusions it supports. - Metric implementation assumptions: Explain why two metrics with the same name can give different assessments, using the contrasting faithfulness definitions as an example.
Why it matters: Checking a metric’s definition prevents apparently comparable scores from hiding different standards. - Mechanical versus judgment scoring: Distinguish automatic metrics from judgment-based metrics, and describe how human or AI judges can supply the evaluative judgment.
Why it matters: The distinction clarifies whether scoring is a mechanical calculation or depends on an assessment of the system’s behavior. - Reference use in metrics: Distinguish reference-based metrics from reference-free metrics and explain why the distinction matters when acceptable answers are unavailable or costly to produce.
Why it matters: The availability of trustworthy reference answers affects which metrics are practical to use. - Execution-based outcome evaluation: Explain what execution-based metrics assess and balance their end-to-end advantages against their infrastructure costs and inability to establish safety on their own.
Why it matters: Achieving a task in a realistic workflow is valuable evidence, but success alone does not show that the system acted appropriately.
Key terms: metric, task, dimension, input, computation method, output, interpretation, precision, faithfulness, DeepEval
3.3 What makes a good evaluation metric
This section introduces the distinction between a measurable metric and one that gives meaningful, reliable insight into AI system performance.
- Measurability versus metric quality: Explain why being measurable is not enough to make a metric useful for evaluating an AI system.
Why it matters: It keeps you from treating an available number as automatically useful evaluation evidence.
Key terms: metric, meaningful or reliable insights, effective metrics
3.3.1 Construct validity
This section explains construct validity, why misleading proxies are risky, and how to check whether a metric measures the quality it is meant to assess.
- Metric and construct alignment: Explain that construct validity is how well a metric captures its intended quality, and distinguish a metric that tracks that quality from a convenient but misleading proxy.
Why it matters: A strong score is useful only if it reflects the outcome you care about. - Construct validity checks: Describe how to check a metric’s construct validity by tracing its connection to the desired outcome, reviewing misleading results in similar contexts, and comparing its scores with human or domain judgments.
Why it matters: These checks help reveal when a better score does not mean a better system.
Key terms: construct validity, construct, proxy, prior experience, human or domain judgments
3.3.2 Causal proximity
The section explains how to judge whether a metric changes for meaningful reasons, how to test that connection, and how feedback from a system’s actions can distort it.
- Meaning of causal proximity: Distinguish causal proximity from construct validity and explain why causal proximity matters when using a metric to improve a system.
Why it matters: A score cannot reliably guide improvements if it changes for reasons unrelated to the improvement. - Context dependence of metric behavior: Explain why the same metric can have strong causal proximity in one setting and weak causal proximity in another, using the medical image example.
Why it matters: The conditions under which a score is collected affect whether its changes can be trusted. - Tests for causal proximity: Describe how controlled quality improvements and irrelevant variations can test whether a metric responds for the right reasons.
Why it matters: Testing both meaningful and irrelevant changes helps reveal whether a metric is useful feedback. - Feedback effect on precision: Trace how acting on churn predictions can lower measured precision even when retention succeeds.
Why it matters: A metric can make a successful intervention look like a worse prediction system.
Key terms: causal proximity, construct validity, feedback loop, precision, data noise, annotation issues, controlled changes, irrelevant variations, feedback effect, customer churn
3.3.3 Movability
This section explains metric movability—whether a metric can still register meaningful improvements—and ways to assess it.
- Meaning of metric movability: Explain that movability is a metric’s ability to register meaningful improvements, including small gains as a system matures.
Why it matters: A metric that cannot show further progress is a poor guide to improving an already capable system. - Checks for lost sensitivity: Describe how score distributions, controlled ablations, and trends across releases can reveal whether a metric still responds to meaningful performance changes.
Why it matters: These checks help tell whether an unchanged score means unchanged performance or an insensitive metric. - Ways to reveal differences: Explain how comparing related metrics and using more challenging inputs can reveal meaningful differences that the current evaluation obscures.
Why it matters: Changing the measure or the difficulty of test cases may expose progress that was previously invisible.
Key terms: movability, headroom, sensitivity, score distribution, saturated, fine-grained ablations, longitudinal trends, metric variants, NDCG, Precision@K
3.3.4 Reliability
This section defines metric reliability, explains why unstable scores are harmful, and gives two ways to assess consistency.
- Metric reliability and reproducibility: Explain that a reliable metric gives similar scores when an evaluation is repeated under the same conditions, and why that consistency matters for interpreting results.
Why it matters: A score is useful for tracking progress only if repeated measurements are reasonably consistent. - Sources of score variation: Describe how judgment-based metrics can give different scores without a meaningful change in system behavior, using the section’s human-rater or LLM-judge examples.
Why it matters: Recognizing these sources of variation helps you avoid mistaking a scoring change for a system change. - Practical reliability checks: Describe how to assess reliability by repeating an evaluation with identical inputs and settings and, for judgment-based metrics, checking agreement across judges.
Why it matters: These checks reveal whether a metric is stable enough to support conclusions from its scores.
Key terms: reliability, reproducibility, human-judgment metrics, LLM-as-a-judge metrics, rubric, temperature, repeat measurements, inter-rater agreement, Cohen’s kappa, Krippendorff’s alpha
3.3.5 Robustness
This section explains what makes an evaluation metric robust, why robustness matters, and how to test it.
- Metric robustness and fragility: Explain that a robust metric remains trustworthy when small, irrelevant aspects of the system, data, or environment change, while a fragile metric shows misleading score swings. Describe why those swings make it harder to trust results and track real progress.
Why it matters: Robustness helps you tell real performance changes from changes caused by the evaluation setup. - Methods for testing robustness: Describe perturbation testing, cross-environment validation, and resampling as ways to check whether metric results hold under controlled minor changes, different evaluation settings, and different samples of data.
Why it matters: These checks reveal whether an apparent result depends on incidental choices in inputs, settings, or data samples.
Key terms: robustness, robust metric, fragile metric, perturbation testing, cross-environment validation, bootstrap, resample evaluation data
3.3.6 Correlation with human judgments
This section explains why metric scores should be checked against human judgments, when that alignment can be misleading, and how to assess it.
- Metric and human judgment alignment: Explain what it means for metric scores to correlate with human judgments and why poor alignment can make optimizing a metric counterproductive.
Why it matters: A rising score is useful only if it reflects an improvement that matters in practice. - Limits of human judgment alignment: Explain why strong correlation with human judgments is not always desirable.
Why it matters: Matching human ratings should not be mistaken for guaranteeing a fair or desirable metric. - Checks against human evaluations: Describe how to check whether an automatic metric can serve as a proxy for human evaluations.
Why it matters: These checks reveal both the overall agreement and specific ways a metric may misjudge outputs.
Key terms: correlation with human judgment, automatic or AI-as-a-judge metrics, human judgments, bias, correlation, paired human evaluations, correlation coefficients, sanity checks, Pearson’s r, Spearman’s ρ
3.3.7 Interpretability and explainability
This section explains why evaluation metrics must be understandable to their users and how to balance intuitive and technical metrics for different purposes.
- Explainable metrics and their value: Explain why a metric must be understandable as well as accurate, especially when evaluation results affect high-stakes decisions.
Why it matters: An impressive number cannot guide a decision if the people using it cannot tell what it means. - Metrics for different evaluation purposes: Distinguish the metrics to prioritize for stakeholder-facing evaluations from those useful for research or diagnosis, and describe how to communicate complex results.
Why it matters: Matching the metric to its audience preserves useful technical detail without leaving decision-makers with an opaque number.
Key terms: explainability, interpretability, accountability, regulatory compliance, operational or stakeholder-facing evaluations, research and diagnostic work, interpretable, intuitive metrics, specialized, technical metrics
3.3.8 Measurability
This section explains whether a metric can be measured in practice and warns against choosing easy-to-measure metrics that miss what matters.
- Practical metric measurability: Explain measurability as the ability to compute a metric easily and reliably within acceptable effort, cost, time, and resource limits. Describe why choosing a metric requires balancing rigor and completeness against practical feasibility.
Why it matters: A metric cannot guide evaluation well if obtaining it reliably exceeds the available resources. - Barriers to metric measurement: Describe how unavailable or poor-quality inputs, reliance on human raters, expensive computation, and ethical or legal limits on data can make a metric difficult to measure.
Why it matters: Recognizing these barriers helps you judge whether a proposed metric can actually be used. - McNamara fallacy in evaluation: Explain how the body-count example illustrates the danger of favoring an easy-to-track metric over harder-to-measure objectives. Apply that warning to selecting AI metrics that reflect the system behavior relevant to a decision.
Why it matters: A convenient metric can make an AI system appear successful without showing whether it meets the evaluation's real objectives.
Key terms: measurability, feasibility, observable inputs, human evaluation, resource-intensive computation, ethical or legal constraints, McNamara Fallacy, poor proxy, body count
3.4 Designing custom metrics
This section explains when existing metrics warrant customization and why abstract evaluation constructs should be broken into observable components.
- Limits of off-the-shelf metrics: Explain when an existing metric may fail to fit an evaluation goal and why adapting it or creating a new metric may be appropriate.
Why it matters: A familiar metric is useful only if its results reflect the distinctions you need to act on. - Observable components of constructs: Describe how to turn an abstract target construct into components that can be observed and assessed.
Why it matters: Observable components make it clearer what a custom metric is assessing.
Key terms: off-the-shelf metrics, granularity, custom metric, domain-specific distinctions, target construct, operationalized, observable components, helpfulness
3.5 Managing and maintaining metrics
This section explains why evaluation metrics need ongoing reassessment and versioning as systems, goals, proxies, and scoring methods change.
- Ongoing metric reassessment: Explain why a metric that once worked may need to be revalidated, revised, retired, or replaced as the system, its use, or the evaluation goal changes.
Why it matters: Rechecking a metric helps prevent yesterday's useful measure from guiding today's decision incorrectly. - Proxy metric breakdown: Describe how the relationship between a proxy metric and the quality it is meant to represent can deteriorate, including when optimizing the proxy fails to improve that quality.
Why it matters: A rising proxy score can conceal stagnant or worsening real-world quality. - Scoring method drift: Explain how scores can drift when a metric's computation or judgments change despite an unchanged metric definition.
Why it matters: A changed score may reflect a changed judging process rather than a changed system. - Metric version traceability: Justify explicitly versioning evolving metrics and recording the version used for every evaluation result.
Why it matters: Without metric versions, an apparent improvement could be only a change in how scores were produced.
Key terms: revalidated, evaluation goals, quality dimensions, proxy breakdown, proxy, underlying construct, lexical overlap, metric drift, judgment guidelines, rubrics
3.6 References
This section lists sources and documentation referenced in the chapter on evaluation metrics.
3.7 Summary
Where this is used later
- (chapter intro) (chapter 4) · The role of metrics in measuring system performance.
- Determining how much evaluation data you need (chapter 4) · Metric movability is responsiveness to genuine changes in system quality.
- AI-as-a-Judge (chapter 5) · Correlation with human judgments is a consideration when evaluating metrics.
- Deciding what judgments should look like (chapter 5) · Designing custom evaluation metrics
Concepts explained in this chapter
- metricA quantitative or scored measure used to assess a system against an evaluation criterion.
- automatic metricA metric computed directly from data by predefined formulas or algorithms without evaluative judgment during computation.
- execution-based metricA metric that assesses system performance through outputs being executed, used, or acted upon in a workflow, application, or environment.
- fidelityA measure of whether a response is faithful to its source information or supporting evidence.
- judgment-based metricA metric whose assessment depends on evaluative judgments involving interpretation, discretion, or normative reasoning.
- reference-based metricA metric that evaluates system outputs by comparing them with one or more reference answers.
- reference-free metricA metric that assesses system outputs without comparing them to a fixed ground truth.
- interpretabilityThe extent to which a system’s behavior, outputs, or evaluation measures can be made understandable to people.
- robustnessThe ability of a system to maintain suitable behavior under varied, unclear, or challenging inputs and conditions.
- LLM-as-a-judgeAn AI system that assesses outputs or behavior according to evaluation criteria.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
