liveBook · Manning

Metric drift

As explained in AI Model Evaluation

Metric drift is a change over time in the relationship between an evaluation metric and the real-world quality or outcome it is intended to represent.

Metric drift occurs when an evaluation metric’s relationship to the real-world quality or outcome it represents changes over time. A metric can move in a promising direction without reflecting a genuine improvement in user experience. Ongoing retraining makes this a concern even after a model passes an A/B test. section 8.2 section 8.7.2

Feedback loops are one cause. A model influences what users see; their behavior produces logs used for later evaluation and retraining. As exposure narrows, offline ranking metrics such as Precision@K or NDCG may correlate less cleanly with online outcomes. Short-term clicks or dwell time can improve while satisfaction, retention, or diversity declines. section 8.2

A movie recommender feedback loop

Suppose a treatment recommends a narrower set of blockbuster movies than a control. Over time, treatment users build more homogeneous watch histories, while control users retain broader histories. The model has changed user behavior as well as recommendations, complicating both evaluation and comparisons between the groups. section 8.2

Monitor drift as part of the evaluation system:

  • Compare rolling aggregates of key online metrics, such as click-through rate and satisfaction, against control baselines over time. section 8.7.2
  • Track input distributions separately using population stability index (PSI) or Kullback–Leibler divergence, with alerts for threshold breaches. These diagnose data drift, not metric drift by themselves. section 8.7.2
  • Connect live changes to offline diagnostics. If offline metrics predict stability while live metrics drift, investigate whether the evaluation framework remains reliable. section 8.7.2
  • Version and monitor any LLM-as-a-judge model and rubric. A judge change can look like product drift even when user experience has not changed. section 8.7.2
Figure 8.5 illustrates a metric drift dashboard mock-up. The left panel shows CTR over time where the stable series fluctuates within normal variance, while the drifting series trends upward beyond the baseline, suggesting feedback-driven distortion. The right panel tracks retention and PSI (Population Stability Index).

The dashboard pairs click-through rate trends with retention and PSI, helping distinguish metric movement from shifts in the input distribution. section 8.7.2

No single online metric establishes real-world quality. Combine online and offline signals with human evaluation and, where appropriate, LLM-as-a-judge evaluation. section 8.0

How AI Model Evaluation covers it

How deep it goes
Explains feedback-driven drift and outlines dashboard monitoring, distribution alerts, and offline diagnostics without specifying an implementation.
Examples
Uses a movie recommender with diverging user histories and a dashboard comparing click-through rate, retention, and PSI.
What you can do after
Diagnose metric drift by comparing live trends with baselines and offline diagnostics while monitoring input distributions and judge versions.
How it connects
Connects metric drift to feedback loops, data drift, offline evaluation, A/B testing, and LLM-as-a-judge evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.