An online metric is a quantitative measure of model or system performance observed in live operation. It can capture technical behavior, such as page load time, or user behavior, such as click-through rate. Together, these measures help connect system performance to product experience. section 4.4.3 section 6.0
Choose metrics for the product outcome you actually want, not just the behavior that is easiest to count. A click can be useful evidence, but it does not by itself establish satisfaction or sustained engagement. A model may improve a local metric while worsening downstream experience or latency. section 8.1.1 section 6.0
Measuring recommendation latency
A movie recommendation test compares a control with five added-latency variants. Average page load time measures the technical change; click-through rate and bounce rate show how user behavior changes. The test is replicated by device type because latency may affect devices differently. section 4.4.3
Table 4.4 Latency Degradation Test Results on Mobile
|
Variant Cell
|
Added Latency
|
Avg Page Load Time
|
CTR
|
Bounce Rate
|
|
Control
|
0ms
|
1050ms
|
23.5%
|
27.8%
|
|
Test Variant 1
|
100ms
|
1155ms
|
22.7%
|
29.1%
|
|
Test Variant 2
|
200ms
|
1250ms
|
20.2%
|
32.7%
|
|
Test Variant 3
|
300ms
|
1360ms
|
16.4%
|
39.5%
|
|
Test Variant 4
|
400ms
|
1455ms
|
12.2%
|
48.3%
|
|
Test Variant 5
|
500ms
|
1552ms
|
8.7%
|
56.9%
|
The mobile results put page load time alongside click-through and bounce rates across the control and latency variants. section 4.4.3
Offline metrics help screen models before launch, but they do not guarantee live impact. Compare offline results with online outcomes over time, logging inference-time signals, outputs, confidence scores, and downstream behavior. The relationship need not be one-to-one: clicks, for example, are influenced by exposure and presentation as well as relevance. section 1.7.2 section 7.6.2
Turn offline findings into live watchpoints using a metric mirror table. For each key offline metric, record the corresponding online measure, its computation, and alert thresholds. Specify category definitions, filters, and missing-data handling; choose aggregation windows and thresholds that account for production variability. section 7.6.2 section 7.6
Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
|
Offline Metric
|
Online Watch point
|
Thresholds
|
Target Value
|
|
Diversity index
|
1−∑pi2 computed daily from top-N categories in live logs
|
Drop > 0.05 vs. control for ≥ 3 days
|
≥ 0.70
|
|
Latency p95
|
p95(model_response_time_ms) aggregated per day
|
> 400ms
|
≤ 350ms
|
|
Bias score by segment
|
Mean(predicted_score[group]) difference between any two groups (absolute value)
|
Gap > 0.10 for any demographic segment
|
≤ 0.05
|
|
Precision@10
|
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
|
Drop > 0.02 vs. control
|
Match or exceed control
|
|
Fairness gap
|
min(segment_score)/max(segment_score)
|
Ratio < 0.85
|
≥ 0.90
|
The mirror table illustrates watchpoints and thresholds for diversity, latency, segment bias, click rate, and fairness. section 7.6.2
Interpret online metrics with their blind spots in mind:
- Exposure can lift engagement without proving greater relevance. Model outputs change user behavior and logged data, which can feed back into later model iterations. section 8.2
- Aggregate outcomes can miss fairness or robustness problems. Use offline diagnostics to identify cohorts and conditions that need explicit online monitoring. section 8.4
- Online metrics are not sufficient alone. Balance them with offline, human, and LLM-as-a-judge evaluations. section 8.0



















