Data drift
Data drift is a change over time in the statistical distribution or characteristics of data relative to a reference distribution or set of characteristics.
Machine Learning System Design
A discrepancy between the statistical characteristics of data used for model development and those of data encountered in the target operating environment.
- How it is used
- Flags changes or mismatches in model inputs that may threaten production performance and fairness.
- How deep it goes
- Provides monitoring choices and alert-design considerations, but stops short of implementing statistical tests.
- Examples
- Contrasts cloudy-weather development images with sunny trial-run images and distinguishes predictable weekend shopping demand from alert-worthy change.
- What you can do after
- Operate/configure drift monitoring by choosing signals, alert thresholds, and possible responses according to label availability and application risk.
Read how Machine Learning System Design explains data drift →
Machine Learning Platform Engineering
Drift detection is the identification of changes in data distributions or related model-serving conditions over time.
- How it is used
- Monitors production data and its relationship with target variables to identify changes that may affect model quality.
- How deep it goes
- Implements microbatch reports with a rolling window, but does not detail the underlying statistical tests.
- Examples
- Uses an income classifier API, and identifies fraud detection, recommendation systems, and image or embedding data as monitoring contexts.
- What you can do after
- Operate microbatch drift monitoring with BentoML and use detected changes to guide model maintenance.
Read how Machine Learning Platform Engineering explains data drift →
The Art of AI Product Development
Domain shift is a change between the data distribution encountered during model operation and the distribution represented by its training data.
- How it is used
- Monitor production inputs and changing data distributions to identify risks and guide prompt adaptation or model updates.
- How deep it goes
- Explains production risks and monitoring responses, without detailing drift detection methods or update procedures.
- Examples
- Uses unpredictable free-text app inputs and demographic imbalances in changing real-world data as examples.
- What you can do after
- Operate continuous monitoring and adapt prompts, examples, or data when production inputs or imbalances change.
Read how The Art of AI Product Development explains data drift →
Reinforcement Learning from Human Feedback
Distribution mismatch is a discrepancy between the probability distribution that generated training data and the distribution of the policy being optimized.
- How it is used
- Corrects off-policy training and distillation when optimization targets differ from the distributions that generated training sequences.
- How deep it goes
- Defines the mismatch, gives the importance-sampling identity, and explains clipping, asynchronous drift, and on-policy distillation.
- Examples
- Uses stale rollout batches, asynchronous inference and training weights, teacher-generated sequences, and student-generated sequences as mismatch cases.
- What you can do after
- Apply it via importance sampling, clipping, sequence-level ratios, or student-sampled reverse-KL distillation.
Read how Reinforcement Learning from Human Feedback explains data drift →
Starting Data Analytics with Generative AI and Python
Domain mismatch is the discrepancy between the data domain used to develop a model and the domain of the data to which it is applied.
- How it is used
- Evaluates whether a pretrained sentiment classifier fits the customer-review data it will analyze.
- How deep it goes
- Defines mismatch through a finance-versus-general-review example and lists risks, but does not establish the model’s actual efficacy.
- Examples
- Contrasts finance-specialized FinBERT-PT-BR with Portuguese customer reviews and flags sarcasm and regional variation.
- What you can do after
- Evaluate whether to proceed with FinBERT-PT-BR by monitoring errors and assessing its efficacy.
Read how Starting Data Analytics with Generative AI and Python explains data drift →
Microsoft Azure in Action
Data drift is a change over time in the distribution or characteristics of input data relative to the data used to train a model.
- How it is used
- Monitors incoming request features and alerts teams when they move outside the model’s training distribution.
- How deep it goes
- Tracks feature changes with examples, but does not specify detection thresholds or remediation steps.
- Examples
- Uses changes in review length and sentiment distribution as examples of shifting incoming features.
- What you can do after
- Operate/configure it by setting alerts for shifts in incoming request features.
Related concepts
- Evidently AIused by · Evidently AI is an open-source Python tool for evaluating, testing, monitoring, and reporting on machine-learning data, models, predictions, and behavior.
- degradationproduces · Degradation is a decline in a system’s or model’s performance, quality, or effectiveness relative to a previous or expected level.
- regressioncontrasts with · A regression is an unintended deterioration in a system’s behavior, performance, or capabilities relative to a prior or expected level, often following a change.
- population stability indexmeasured by · The population stability index (PSI) is a measure of the difference between two probability distributions, often used to quantify shifts in a population or model inputs.
- monitoringmeasured by · Monitoring is the ongoing observation and measurement of a system’s behavior, state, or operational signals to assess its condition and detect changes or problems.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.