A watchpoint is a live monitoring check for a known model weakness or risk. It runs alongside an experiment’s primary success metrics, using an alert threshold to flag problems early enough to investigate or intervene. section 7.6 section 7.6.1
Define watchpoints before an A/B test begins:
- Identify failure modes from offline evaluation or sandbox testing, such as segment-level gaps, popularity bias, or latency spikes. section 7.1 section 7.6.1
- Choose an online measurement that tracks the corresponding risk, including the affected user segments where relevant. section 7.6.1
- Set an alert threshold tied to the offline finding and historical production variability. Specify what happens if it is crossed, such as investigation, pausing, rollback, or a patched variant. section 7.6.1 section 7.6.2
- Compare offline and online results over time to learn which offline metrics predict live performance. section 7.6
Repeated recommendations
A ranking model repeatedly placed Batman first, even for users without relevant watch history. A watchpoint can track repeat-title frequency in top-N recommendations by user segment: count unique titles in the top five slots per session and track the proportion of sessions where the same title appears first for more than three consecutive sessions. Flag a segment if its repeat-title frequency exceeds control by more than 15% for two consecutive days. The alert can prompt a pause, rollback, or patched variant. section 7.6.1
A metric mirror table links each key offline metric to its online watchpoint, production computation, and alert threshold. Keep definitions consistent, including categories, filters, and missing-data handling, and choose an aggregation cadence fast enough to detect deterioration. An online measure may still be a proxy: top-10 click rate, for example, is influenced by exposure and presentation and is not equivalent to offline Precision@10. section 7.6.2
Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
|
Offline Metric
|
Online Watch point
|
Thresholds
|
Target Value
|
|
Diversity index
|
1−∑pi2 computed daily from top-N categories in live logs
|
Drop > 0.05 vs. control for ≥ 3 days
|
≥ 0.70
|
|
Latency p95
|
p95(model_response_time_ms) aggregated per day
|
> 400ms
|
≤ 350ms
|
|
Bias score by segment
|
Mean(predicted_score[group]) difference between any two groups (absolute value)
|
Gap > 0.10 for any demographic segment
|
≤ 0.05
|
|
Precision@10
|
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
|
Drop > 0.02 vs. control
|
Match or exceed control
|
|
Fairness gap
|
min(segment_score)/max(segment_score)
|
Ratio < 0.85
|
≥ 0.90
|
Compare example watchpoint formulas, alert thresholds, and target values for diversity, latency, bias, precision, and fairness. section 7.6.2
