liveBook · Manning

Metric

As explained in AI Model Evaluation

A quantitative measure of model performance derived from behavior or outcomes observed during live operation.

An online metric is a quantitative measure of model or system performance observed in live operation. It can capture technical behavior, such as page load time, or user behavior, such as click-through rate. Together, these measures help connect system performance to product experience. section 4.4.3 section 6.0

Choose metrics for the product outcome you actually want, not just the behavior that is easiest to count. A click can be useful evidence, but it does not by itself establish satisfaction or sustained engagement. A model may improve a local metric while worsening downstream experience or latency. section 8.1.1 section 6.0

Measuring recommendation latency

A movie recommendation test compares a control with five added-latency variants. Average page load time measures the technical change; click-through rate and bounce rate show how user behavior changes. The test is replicated by device type because latency may affect devices differently. section 4.4.3

Table 4.4 Latency Degradation Test Results on Mobile
Variant Cell
Added Latency
Avg Page Load Time
CTR
Bounce Rate
Control
0ms
1050ms
23.5%
27.8%
Test Variant 1
100ms
1155ms
22.7%
29.1%
Test Variant 2
200ms
1250ms
20.2%
32.7%
Test Variant 3
300ms
1360ms
16.4%
39.5%
Test Variant 4
400ms
1455ms
12.2%
48.3%
Test Variant 5
500ms
1552ms
8.7%
56.9%

The mobile results put page load time alongside click-through and bounce rates across the control and latency variants. section 4.4.3

Offline metrics help screen models before launch, but they do not guarantee live impact. Compare offline results with online outcomes over time, logging inference-time signals, outputs, confidence scores, and downstream behavior. The relationship need not be one-to-one: clicks, for example, are influenced by exposure and presentation as well as relevance. section 1.7.2 section 7.6.2

Turn offline findings into live watchpoints using a metric mirror table. For each key offline metric, record the corresponding online measure, its computation, and alert thresholds. Specify category definitions, filters, and missing-data handling; choose aggregation windows and thresholds that account for production variability. section 7.6.2 section 7.6

Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
Offline Metric
Online Watch point
Thresholds  
Target Value
Diversity index
1−∑pi2 computed daily from top-N categories in live logs
Drop > 0.05 vs. control for ≥ 3 days
≥ 0.70
Latency p95
p95(model_response_time_ms) aggregated per day
> 400ms
≤ 350ms
Bias score by segment
Mean(predicted_score[group]) difference between any two groups (absolute value)
Gap > 0.10 for any demographic segment
≤ 0.05
Precision@10
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
Drop > 0.02 vs. control
Match or exceed control
Fairness gap
min(segment_score)/max(segment_score)
Ratio < 0.85
≥ 0.90

The mirror table illustrates watchpoints and thresholds for diversity, latency, segment bias, click rate, and fairness. section 7.6.2

Interpret online metrics with their blind spots in mind:

  • Exposure can lift engagement without proving greater relevance. Model outputs change user behavior and logged data, which can feed back into later model iterations. section 8.2
  • Aggregate outcomes can miss fairness or robustness problems. Use offline diagnostics to identify cohorts and conditions that need explicit online monitoring. section 8.4
  • Online metrics are not sufficient alone. Balance them with offline, human, and LLM-as-a-judge evaluations. section 8.0

How AI Model Evaluation covers it

How deep it goes
Explains metric selection, offline alignment, monitoring formulas, alert thresholds, and interpretation pitfalls with a worked recommendation experiment.
Examples
Uses a movie recommendation latency test and a metric mirror table with click-rate, diversity, latency, and fairness watchpoints.
What you can do after
Evaluate and understand live model impact by selecting product-aligned metrics, defining watchpoints, and interpreting proxy limitations.
How it connects
Connects online metrics to A/B testing, offline evaluation, feedback loops, exposure bias, fairness, and robustness.

Also explained in

Compare how 20 books explain metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.