liveBook · Manning

Watchpoint

As explained in AI Model Evaluation

A live monitoring check that tracks specified online behavior, model weaknesses, or risks and triggers investigation or intervention when needed.

A watchpoint is a live monitoring check for a known model weakness or risk. It runs alongside an experiment’s primary success metrics, using an alert threshold to flag problems early enough to investigate or intervene. section 7.6 section 7.6.1

Define watchpoints before an A/B test begins:

  • Identify failure modes from offline evaluation or sandbox testing, such as segment-level gaps, popularity bias, or latency spikes. section 7.1 section 7.6.1
  • Choose an online measurement that tracks the corresponding risk, including the affected user segments where relevant. section 7.6.1
  • Set an alert threshold tied to the offline finding and historical production variability. Specify what happens if it is crossed, such as investigation, pausing, rollback, or a patched variant. section 7.6.1 section 7.6.2
  • Compare offline and online results over time to learn which offline metrics predict live performance. section 7.6

Repeated recommendations

A ranking model repeatedly placed Batman first, even for users without relevant watch history. A watchpoint can track repeat-title frequency in top-N recommendations by user segment: count unique titles in the top five slots per session and track the proportion of sessions where the same title appears first for more than three consecutive sessions. Flag a segment if its repeat-title frequency exceeds control by more than 15% for two consecutive days. The alert can prompt a pause, rollback, or patched variant. section 7.6.1

A metric mirror table links each key offline metric to its online watchpoint, production computation, and alert threshold. Keep definitions consistent, including categories, filters, and missing-data handling, and choose an aggregation cadence fast enough to detect deterioration. An online measure may still be a proxy: top-10 click rate, for example, is influenced by exposure and presentation and is not equivalent to offline Precision@10. section 7.6.2

Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
Offline Metric
Online Watch point
Thresholds  
Target Value
Diversity index
1−∑pi2 computed daily from top-N categories in live logs
Drop > 0.05 vs. control for ≥ 3 days
≥ 0.70
Latency p95
p95(model_response_time_ms) aggregated per day
> 400ms
≤ 350ms
Bias score by segment
Mean(predicted_score[group]) difference between any two groups (absolute value)
Gap > 0.10 for any demographic segment
≤ 0.05
Precision@10
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
Drop > 0.02 vs. control
Match or exceed control
Fairness gap
min(segment_score)/max(segment_score)
Ratio < 0.85
≥ 0.90

Compare example watchpoint formulas, alert thresholds, and target values for diversity, latency, bias, precision, and fairness. section 7.6.2

How AI Model Evaluation covers it

How deep it goes
Details failure-mode selection, online measurement, alert thresholds, formula consistency, aggregation cadence, and intervention options.
Examples
Uses repeated movie recommendations and a mirror table covering diversity, latency, bias, precision, and fairness.
What you can do after
Operate/configure watchpoints with measurements and thresholds that prompt investigation or intervention during an experiment.
How it connects
Connects offline evaluation, A/B testing, model selection, metric mirror tables, and online monitoring.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.