A production IT support assistant can lose accuracy even when its weights have not changed. New platforms, revised procedures, and changing user language can make its answers less useful or wrong. Drift is change in inputs, correct answers, phrasing, or model behavior that can erode performance over time. section 9.0 section 9.2
Distinguish four forms of drift before deciding how to respond:
- Data drift: the input distribution changes. Questions about Kubernetes networking may grow from a rarely represented topic to 40% of requests. section 9.2
- Concept drift: the correct response changes for the same input. A password reset request may now require an SSO provider rather than an Active Directory portal. section 9.2
- Prompt drift: users change how they phrase requests, moving from formal questions to shorthand such as “vpn broken pls help.” section 9.2
- Model drift: model behavior changes, as when an API provider updates the underlying model. Changed evaluation data or a retrained version can also reveal regressions. section 9.2
Monitor both input distributions and behavior. A TF-IDF cosine-centroid detector can flag distribution-level changes, while periodic canary prompts check for behavioral degradation. Calibrate thresholds on your own data; use the train/validation split to establish a baseline rather than treating default values as final. section 9.11
Make response part of the deployment lifecycle. Retrain when outcomes warrant it, starting from the most recent checkpoint, and run the full evaluation suite before deployment. Keep a registry of model and data versions and the previous two model versions ready for rollback. Assign monitoring responsibilities: evaluation sets, drift flags, and gates require ongoing engineer and reviewer time. section 9.11 section 1.3

















