A model can keep serving predictions while becoming less useful. Model drift, also called model decay or staleness, is declining performance over time as incoming data or real-world relationships change. It may appear suddenly or gradually, so a working API and clean data are not enough to establish that predictions remain reliable. section 14.4
Separate two common sources of drift:
- Data drift, also called feature drift or covariate shift, means incoming inputs differ substantially from training inputs. A model may still work on familiar cases but perform poorly on a new population. section 14.4.1
- Concept drift means learned relationships no longer hold: the same kinds of inputs may now call for different expected outputs, even when feature distributions look similar. section 14.4 section 14.6
Figure 14.7 With a concept drift, identical inputs may lead to new expected outputs; with a data drift, incoming data changes while the model is not adapted.
Compare what changes: incoming inputs in data drift, or expected outputs for identical inputs in concept drift. section 14.4
A new customer segment
A purchase-prediction model trained mostly on existing marketplace users is applied to users acquired through a different advertising campaign. If that source was scarce in training data, weak performance on the new segment may indicate data drift. Monitor results by segment rather than relying only on an overall score. section 14.4.1 section 14.4.3
Monitor signals that answer different questions:
- Compare quality metrics with historical baselines once ground truth or feedback arrives; use measures appropriate to the task, such as mean absolute error, accuracy, or ranking metrics. section 14.4 section 14.4.3
- Track input feature statistics and prediction distributions. When labels are delayed, input and prediction drift can provide earlier proxy signals; neither alone establishes that quality has fallen. section 14.4.3
- Adjust monitoring granularity to risk. Critical models may need segment-level and fairness metrics; lower-risk models may need only standard task metrics. section 14.4.3
Investigate before pressing retrain:
- Check data quality first, including entry errors, schema changes, and upstream models. A broken input pipeline calls for a different fix from a genuine shift in the population. section 14.4.4
- For genuine input changes, plot affected feature distributions and consult domain experts about the cause. Then assess performance and consider new features, retraining on updated data, or a model for the new segment. section 14.4.4 section 14.4.1
- If learned relationships have changed, consider retraining with all available data, weighting recent data more heavily, or dropping older data when enough new data exists. section 14.6
Keep inventory predictions available
An inventory system monitors feature drift and data quality. If its feature-dependent model becomes unsuitable, it can switch to a less externally dependent time-series model; as a last resort, it uses the previous week’s sales adjusted for expected events and holidays. Fallbacks preserve an operating path while the drift is investigated. section 13.5.1
External feature extractors add another dependency: an unversioned update can change embeddings unexpectedly, while leaving the external model untouched can make it less useful as local data shifts. Validate and monitor external models as part of the feature pipeline. section 11.1.3
