Model drift is a change in a model’s data, relationships, or operating conditions that can make it less suitable over time. A performance problem may reveal that the model was inadequate from the start, or that the world around it has changed. section 10.2.2
Drift can point to different problems:
- Poor testing or model selection can expose weaknesses in the original production model. section 10.2.2
- Concept drift occurs when relationships that once held no longer apply. section 10.2.2
- Changes in data noise can increase errors, including error types that were previously unimportant. section 10.2.2
A changing shopping pattern
A model learns that customers often buy smart trousers and shirts together. If a shift to working from home removes demand for smart trousers, its recommendations may become less effective until the model can adapt. section 10.2.2
When drift is detected, retraining means building a new model to address the production model’s issues. It requires a modeling approach, a training set that captures the changed domain, a testing set or regime, and stakeholder buy-in. Detect shifts early: enough data to characterize them or train and test a replacement may not yet be available, and deployment timing can affect the business. section 10.2.3
