liveBook · Manning

Chapter 8: Pitfalls of online metrics

What you will learn in chapter 8 of AI Model Evaluation

The chapter introduction argues that trustworthy AI model evaluation requires more than online metrics alone, while previewing pitfalls discussed later.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction argues that trustworthy AI model evaluation requires more than online metrics alone, while previewing pitfalls discussed later.

Key terms: online metrics, Balanced evaluations, offline, online, human, LLM-as-a-judge

8.1 Choosing metrics that actually matter

This section explains why choosing online metrics for AI models is consequential and why the wrong metrics can misrepresent a model’s effect.

Key terms: online metrics, offline diagnostics, A/B test, probabilistic, adaptive, long-tailed, segment-specific

8.1.1 Weak metrics waste the offline investment

This section shows how an online metric can undermine extensive offline evaluation when it measures something other than the product’s real goal.

Key terms: online metric, click-through rate, CTR, north star, long-term viewing satisfaction, signal, noise, product value

8.1.2 Contract between product strategy and AI model evaluation

This section frames online metrics as a contract that connects product strategy to model evaluation through success metrics, guardrails, and explicit trade-offs.

Key terms: evaluation contract, product strategy, primary metric, guardrail metrics, non-inferiority margins, superiority, guardrails, CTR, treatment, control

8.2 Feedback loops unique to AI models

This section explains how model-driven feedback loops can distort online and offline evaluation, and why evaluations need safeguards against apparent short-term gains that harm the product over time.

Key terms: feedback loop, logged data, exposure bias, reinforced errors, impact masking, CTR, counterfactual logging bias, offline metric drift, logging policy, Precision@K

8.2.1 How to combat feedback loops in evaluations

This section explains how exploration, stable baselines, offline checks, and metric guardrails help evaluations distinguish genuine improvement from feedback-driven gains.

Key terms: exploration, multi-armed bandits (MABs), propensity, counterfactual evaluations, long-term holdout, stable baseline model, feedback-driven distribution shift, quarterly holdbacks, inverse propensity scoring (IPS), rolling dataset

8.3 Metric gaming and model exploitation

This section explains how an AI model can improve an evaluation metric while failing to improve the product outcome that metric is meant to represent.

Key terms: metric gaming, reward hacking, objective function, imperfect proxy, product outcome

8.3.1 Reward hacking in AI models

This section explains how models can improve a measured online metric while harming the product outcome, and why evaluation needs checks beyond the primary metric.

Key terms: reward hacking, measurable lever, CTR, agentic systems, task completion, online metric, primary metric, guardrails, non-inferiority margins, richer outcome metrics

8.3.2 Reward hacking isn’t limited to models

This subsection shows that users and content creators can exploit engagement metrics, so evaluations must watch for adversarial behavior beyond the primary outcome.

Key terms: reward hacking, engagement signals, adversarial prompts, secondary signals, primary outcomes, online metric contract

8.3.3 Why this risk is sharper in AI A/B tests

Explains why AI models are more prone to metric gaming in A/B tests than static UX changes, and why the test metric alone may not reveal it.

Key terms: active optimizers, metric gaming, future training data, success metrics, guardrails, NIMs, offline diagnostics

8.3.4 Combating reward hacking and exploitation

This section describes ways to make evaluation targets harder to exploit and detect reward hacking before or during online testing.

Key terms: multi-metric contracts, guardrails, NIMs, robust outcome metrics, adversarial offline checks, synthetic or adversarial data, segment analysis, cohorts, exploration traffic, feedback loops

8.4 The blind spots of online metrics

This section explains why top-line A/B test metrics can miss important AI model qualities and how offline evaluations can guide targeted online monitoring.

Key terms: online metrics, primary outcome metrics, blind spots, offline evaluation strategy, diagnostics, stress tests, targeted cohort analyses

8.4.1 Fairness across segments

Shows how cohort-level monitoring and guardrails can prevent healthy A/B test averages from hiding widening fairness gaps.

Key terms: offline cohort analysis, online monitoring, stratify, non-inferiority margins, subgroup-specific guardrails

8.4.2 Robustness in edge cases

This section explains why A/B tests can miss edge-case failures and how isolated shadow testing can make those failures measurable before a model is shipped.

Key terms: offline stress tests, edge cases, online A/B tests, statistically significant, robustness, shadow testing, non-serving model, rubric, long-tail query set, prompt class

8.4.3 Qualities like trust, coherence, or novelty that resist simple measurement

This section shows why healthy top-line metrics can hide declining trust, coherence, and novelty, and identifies behavioral signals that make those qualities easier to monitor.

Key terms: trust, CTR, dwell time, retention, negative feedback signals, skip streaks, session abandonment, cohort-level trend, trust debt, coherence

8.5 The importance of the right metrics framework for AI

This section explains why AI online evaluation needs an aligned metrics framework and how a metric hierarchy connects offline diagnostics with online results.

Key terms: primary metric, proxy, metrics framework, metric hierarchies, offline layer, diagnostic tier, proxy and outcome tiers

8.5.1 Building metric hierarchies

This section explains how a three-layer metric hierarchy connects model behavior to product success and exposes misleading online wins.

Key terms: metric hierarchy, outcome metrics, proxy metrics, diagnostic metrics, 30-day retention, CTR, watch-time per session, genre diversity index, fairness gaps by cohort

8.5.2 Revisiting metrics as models and products evolve

This section explains why evaluation metrics must change as models mature, product goals shift, and new model designs introduce new failure modes.

Key terms: metrics framework, proxies, outcome alignment, versioning metrics, proxy validity, historical correlation studies, rolling datasets, diagnostics, CTR, factuality

8.5.3 Aligning evaluation targets with both product and model goals

This section explains how a metrics framework connects product goals to model behavior so online metrics indicate whether the model improves the product.

Key terms: outcome metrics, guardrail metrics, proxies, diagnostics, metrics framework

8.6 Statistical refinements for AI online metrics

This section explains why raw A/B test differences can mislead for AI models and how variance reduction can make subtle effects easier to detect.

Key terms: heterogeneous lift, user segment, overall average, covariate adjustment, CUPED, regression adjustment, variance, power

8.6.1 Top-line averages may misrepresent specific user groups

This section explains why aggregate A/B metrics can hide uneven model effects or be noisy, and why analysis should follow the product’s target population.

Key terms: top-line averages, heterogeneous effects, subgroup gains, user-level variance, watch time, binge users, target population, average effects, metric slicing, variance reduction

8.6.2 Variance reduction techniques (CUPED, regression adjustment)

Explains how CUPED can make a real treatment effect detectable by removing predictable variation, and why its covariate must come from before treatment.

Key terms: CUPED, covariate adjustment, residual variance, treatment effect, confidence interval, pre-experiment covariate, treatment assignment, randomization

8.6.3 Pitfalls of covariate choices

This section explains how poorly chosen covariates can undermine online treatment estimates and how to check whether an adjustment is trustworthy.

Key terms: post-treatment leakage, covariate, randomization, treatment effect, weak covariates, correlation, CUPED, outcome variance, bias amplification, structural differences

8.7 Engineering considerations

This section warns that production infrastructure can distort AI evaluation metrics through logging, attribution, and monitoring quirks.

Key terms: logging, attribution, monitoring, evaluation metrics

8.7.1 Logging and attribution in ensemble or blended models

This section explains how granular logs make online metric changes attributable to the components of blended models and agentic systems.

Key terms: attribution, per-component scores, decision trace, final blended score, task completed, retrieved context, tool response, final outcome

8.7.2 Monitoring for metric drift in retraining pipelines

The section explains how to monitor retrained models for data and metric drift, and how to avoid mistaking changes in an evaluation judge for changes in the product.

Key terms: data drift, metric drift, retraining, online metrics, metric drift dashboards, rolling aggregates, control baselines, population stability indexes (PSI), Kullback–Leibler divergence, offline diagnostics

8.7.3 Alert fatigue

This subsection explains how to keep guardrail alerts actionable without overwhelming teams with noise.

Key terms: alert fatigue, guardrail metrics, non-inferiority margins (NIMs), guardrails, severity, experiment metadata, auditability

8.7.4 Guarding against variance inflation

This section explains how feedback loops can inflate variance in AI model A/B tests and presents three engineering tactics for keeping online evaluations trustworthy.

Key terms: variance inflation, feedback loops, treatment and control populations, stratified analysis, stable cohorts, treatment–control differences, exploration traffic, randomized exposure, unbiased counterfactual baseline, long-term holdbacks

8.8 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 7: From offline evaluation to live experiment Chapter 9: LLM-as-a-judge fundamentals →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.