A model can work on development data and struggle with the inputs it encounters in use. Differences in where data comes from, how it is preprocessed, or what is filtered out can create a distribution mismatch that affects performance and fairness. The mismatch may be hard to detect before launch, especially when live inputs are not yet available or the model itself changes which data is collected. section 6.6
Cloudy training scenes, sunny trial run
An augmented-reality team collected and tested most of its camera data in cloudy St. Petersburg. Its prerelease model performed poorly when investors tried it in sunny Cyprus. The problem was a mismatch between development images and real-world scenes—a training-serving skew closely related to data drift. Collecting and labeling trial-run data, or adapting the model with it, can help address that mismatch. section 14.3.5
Before adding drift alerts, decide what action an alert would trigger. Predictable patterns, such as higher loungewear demand on weekends, should be accounted for rather than treated as surprises. Set thresholds around changes that matter to the model and the real-world process behind its data. section 14.4.4
Choose monitoring to fit the available evidence:
- If true labels arrive promptly and model quality can be measured directly, focus on quality and use rule-based checks for broken inputs; distribution-drift monitoring may be unnecessary. section 14.4.4
- For a critical system with delayed ground truth and interpretable features, consider a drift dashboard and statistical tests for distribution and feature-correlation changes. Give known, business-critical features more attention. section 14.4.4
- Plan the response as well as the detection: monitoring a core model’s input features can help trigger a switch to a fallback, whether that switch is automatic, manual, or hybrid. section 13.4
