Data drift is a change in the distribution of inputs a model encounters over time. A model can lose accuracy even when its weights and training data have not changed: production traffic has moved away from the inputs it learned to handle. section 9.2
IT support after a platform migration
An IT support model trained largely on VPN and email questions encounters a new cloud platform. Six months later, 40% of incoming questions concern Kubernetes networking, a topic barely represented in training. The input shift is a warning that the model’s original coverage may no longer match demand. section 9.2
Data drift is distinct from concept drift, where the correct answer changes, and model drift, where model behavior changes, such as after an API provider update. Prompt drift describes changes in how users phrase requests, such as adopting conversational shorthand. section 9.2
A similar distribution mismatch can exist at launch: training and validation examples may be clean and curated while production queries contain typos, incomplete sentences, or mixed languages. Validation metrics can look healthy if the validation set shares the training set’s bias. section 6.8.3
Reduce mismatch and maintain production quality:
- Sample training examples from real production traffic where possible. Include ambiguous, incomplete, out-of-scope, and typo-containing requests; validate on a separate held-out set drawn from production. section 6.8.3
- Anchor synthetic examples to real seed examples so their style and vocabulary do not introduce a separate shift. Treat a 30% synthetic-data cap as a first-run heuristic, not a universal threshold. section 3.4.3
- Monitor input distributions and periodically run canary prompts and evaluations. Calibrate drift thresholds against your own data; a detector alert is a leading indicator, not proof that retraining is needed. section 9.11 section 9.7.1
- Investigate alerts using canary results, user feedback, and recent outputs. If meaningful degradation is confirmed, sample recent traffic, add targeted examples, retrain, and compare against the previous baseline. section 9.7.1 section 3.5.3
Schedule evaluations with current queries, for example quarterly, rather than waiting for users to notice degradation. Retraining carries ongoing costs; use simpler approaches when they meet the quality bar, and gate retrained versions through the evaluation suite before deployment. section 5.10 section 6.13.1 section 9.11










