Prediction drift is a significant change over time in a model’s predictions. It provides an early reliability signal when ground-truth labels arrive too late for immediate quality monitoring. section 14.4.3
Use prediction drift to support operational monitoring:
- Detect changes with statistical tests, probability-distance metrics, or shifts in descriptive statistics for model outputs. section 14.4.3
- Treat it as a proxy when ground truth is delayed, then confirm model quality with metrics such as RMSE, MAE, accuracy, precision, or F1. section 14.4.3
- Keep prediction drift distinct from input data drift: output changes warn about predictions, while input drift tracks feature changes. section 14.4.3
Retail forecasting alerts
A retail forecasting system receives sales labels 15 minutes after they occur. It monitors next-day, next-week, and next-month predictions, testing Population Stability Index above 0.2 and Wasserstein distance above 0.1. When comparing April 2021 with April 2022, it scales the control predictions by the expected 15% business growth before calculating drift. section 14.5.1
