liveBook · Manning

Data drift

Also called distribution shift, distributional shift, domain shift

Data drift is a change in the statistical distribution or characteristics of data relative to a reference, such as data used to train a model.

How 11 Manning books explain it

Machine Learning System Design

A discrepancy between the statistical characteristics of data used for model development and those of data encountered in the target operating environment.

How it is used
Guides data consistency and monitoring decisions across development and production.
How deep it goes
Outlines causes, monitoring choices, alert thresholds, and responses without deriving statistical tests.
Examples
Uses a camera system tested in different climates and a loungewear forecast with recurring weekend demand.
What you can do after
Evaluate and design drift monitoring around label availability, feature importance, alert thresholds, and possible responses.

Read how Machine Learning System Design explains data drift →

AI Model Evaluation

Feature drift detection is the identification of changes over time in the distributions or behavior of model input features.

How it is used
Supports production observability and A/B test readiness by flagging changes in model inputs that can undermine offline-to-online performance.
How deep it goes
Defines feature drift and demonstrates KS and PSI checks with alerts and example thresholds, while cautioning that shifts alone do not establish harm.
Examples
Uses changing remote-job preferences and seasonal shifts in user behavior to illustrate changing inputs.
What you can do after
Diagnose/debug with KS and PSI checks, then investigate flagged shifts using fresh performance data and segment analysis.

Read how AI Model Evaluation explains data drift →

Quantization and Fast Inference

Distribution mismatch is a difference between the value distribution represented by calibration data and that encountered in production.

How it is used
Identifies calibration-to-production mismatch as an accuracy risk in post-training activation quantization.
How deep it goes
Explains range estimation and clipping, then provides a sampling and layer-wise validation workflow.
Examples
Uses sentiment analysis, image conditions, language-model prompts, and a per-layer calibration validation report.
What you can do after
Diagnoses calibration coverage gaps and revises sampling to better represent production activations.

Read how Quantization and Fast Inference explains data drift →

Machine Learning Platform Engineering

Drift detection is the identification of changes in data distributions or related model-serving conditions over time.

How it is used
Monitors production data and its relationship with target variables to identify changes that may affect model quality.
How deep it goes
Implements microbatch reports with a rolling window, but does not detail the underlying statistical tests.
Examples
Uses an income classifier API, and identifies fraud detection, recommendation systems, and image or embedding data as monitoring contexts.
What you can do after
Operate microbatch drift monitoring with BentoML and use detected changes to guide model maintenance.

Read how Machine Learning Platform Engineering explains data drift →

LLM Customization and Fine-Tuning

A change over time in the characteristics or distribution of data a model encounters, potentially reducing its performance.

How it is used
Frames changing production inputs as a maintenance risk for fine-tuned models.
How deep it goes
Defines input drift, distinguishes related drift types, and gives monitoring and retraining decision steps.
Examples
Illustrates drift through IT support questions shifting toward Kubernetes networking after a cloud migration.
What you can do after
Diagnose input drift with a TF-IDF cosine-centroid detector and canary prompts, then investigate whether output degradation warrants retraining.

Read how LLM Customization and Fine-Tuning explains data drift →

The Art of AI Product Development

Domain shift is a change between the data distribution encountered during model operation and the distribution represented by its training data.

How it is used
Monitor production inputs and changing data distributions to identify risks and guide prompt adaptation or model updates.
How deep it goes
Explains production risks and monitoring responses, without detailing drift detection methods or update procedures.
Examples
Uses unpredictable free-text app inputs and demographic imbalances in changing real-world data as examples.
What you can do after
Operate continuous monitoring and adapt prompts, examples, or data when production inputs or imbalances change.

Read how The Art of AI Product Development explains data drift →

Timeless Algorithms, The Foundational Ideas

A distribution shift is a change in the input data distribution between the conditions under which a model was trained and those under which it is evaluated or applied.

How it is used
Frames changes between training data and production conditions as a source of model failure that calls for monitoring.
How deep it goes
Defines the train-to-production mismatch and traces causes and failure modes, but does not prescribe a detection procedure.
Examples
Uses a pedestrian detector trained on clear daytime images to illustrate failures at night, in severe weather, or with obstructions.
What you can do after
Diagnose production risk by monitoring model performance as operating conditions change.

Read how Timeless Algorithms, The Foundational Ideas explains data drift →

Reinforcement Learning from Human Feedback

Distribution mismatch is a discrepancy between the probability distribution that generated training data and the distribution of the policy being optimized.

How it is used
Corrects policy-optimization estimates and motivates training on student-generated sequences in distillation.
How deep it goes
Derives importance-sampling correction and contrasts token- with sequence-level ratios, including clipping failure modes.
Examples
Uses stale-policy rollouts, asynchronous model weights, student-generated completions, and shifted task rules as examples.
What you can do after
Apply importance sampling to reweight samples from an older policy when estimating updates for the current policy.

Read how Reinforcement Learning from Human Feedback explains data drift →

Starting Data Analytics with Generative AI and Python

Domain mismatch is the discrepancy between the data domain used to develop a model and the domain of the data to which it is applied.

How it is used
Flags the risk of applying a financial-text classifier to general customer reviews.
How deep it goes
Lists likely drawbacks and monitoring options but does not establish the model’s actual performance on the reviews.
Examples
Contrasts FinBERT-PT-BR’s financial focus with Portuguese customer reviews and demonstrates positive-review filtering.
What you can do after
Evaluate a classifier’s suitability for customer reviews and consider human review and error monitoring.

Read how Starting Data Analytics with Generative AI and Python explains data drift →

Delivering Machine Learning Projects: From design to deployment

Feature drift is a change over time in the characteristics or meaning of the features supplied to a machine learning model.

How it is used
Protects deployed model performance by prompting teams to monitor inputs and behavior for change.
How deep it goes
Explains domain, noise, and interface causes, then recommends monitoring and retesting without detailing detection algorithms.
Examples
Uses changing noise levels and types, plus subtle input changes after an API update, as drift examples.
What you can do after
Diagnose/debug with it by monitoring model behavior and inputs, setting assertions, and retesting after changes.

Read how Delivering Machine Learning Projects: From design to deployment explains data drift →

Microsoft Azure in Action

Data drift is a change over time in the distribution or characteristics of input data relative to the data used to train a model.

How it is used
Monitors incoming request features and alerts teams when they move outside the model’s training distribution.
How deep it goes
Tracks feature changes with examples, but does not specify detection thresholds or remediation steps.
Examples
Uses changes in review length and sentiment distribution as examples of shifting incoming features.
What you can do after
Operate/configure it by setting alerts for shifts in incoming request features.

Read how Microsoft Azure in Action explains data drift →

Also covered in

What to know first

Related concepts

Questions about data drift

What is data drift?

Data drift is a change in the statistical distribution or characteristics of data relative to a reference, such as data used to train a model.

Which Manning books explain data drift?

Machine Learning System Design, AI Model Evaluation, Quantization and Fast Inference, Machine Learning Platform Engineering, LLM Customization and Fine-Tuning, The Art of AI Product Development, Timeless Algorithms, The Foundational Ideas, Reinforcement Learning from Human Feedback and 3 more.

What should I know before learning data drift?

Manning books that teach data drift usually introduce accuracy, availability, bias and latency first.

What is the difference between data drift and regression?

Data drift is a change in the statistical distribution or characteristics of data relative to a reference, such as data used to train a model. A regression is an unintended deterioration or loss of previously established behavior, performance, or capability following a change.

What is data drift also called?

Data drift is also called distribution shift, distributional shift or domain shift.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.