A model can perform well at evaluation and then encounter different data in production. Data drift is a change in the distribution or properties of data relative to the data used for training. If the model’s earlier assumptions no longer hold, predictions may become less reliable. Detecting drift is therefore part of production monitoring, alongside performance, errors, and operational metrics; a drift alert signals a change to investigate, not proof that every prediction is wrong. section 6.2 section 2.1.6
Choose what to compare based on the data and the change of concern:
- For tabular inputs, compare each feature’s distribution in a current inference dataset with a reference dataset from training. Covariate shift describes changing input-feature distributions; changes in labels or class frequencies are related shifts worth monitoring separately. section 6.2 section 6.2.1
- For images, compare properties such as brightness, contrast, and aspect ratio between training and inference images. For a recommender, changes in ratings may be investigated through the distributions of learned user and item factors. section 11.2.1 section 11.2.2
With Evidently, create a DataDriftPreset report, run it against reference and current data frames, and save the result as HTML. Its feature rows show the two distributions, statistical test, and drift score. In the illustrated report, Wasserstein distance uses a default drift threshold of 0.1. A single report describes one snapshot; repeated reports in a dashboard show how results change over time. section 6.2.1
Listing 6.5 Generating a data drift report using DataDriftPreset
report = Report(metrics=[
DataDriftPreset(),
]) #1
report.run(
reference_data=reference_data,
current_data=current_data
) #2
report.save_html("drift_report.html") #3
The listing shows the minimal report workflow: configure DataDriftPreset, supply reference and current data, then save the report. section 6.2.1
Place the comparison where predictions are made:
- In batch inference, put a Kubeflow Pipelines drift component after feature retrieval and before inference. Retrieve the model’s reference data through its MLflow run, compare it with current features, publish the report to Evidently, and stop pipeline execution if drift is detected. section 6.2.2
- For a BentoML prediction API, accumulate requests into a moving microbatch before testing. Comparing a reference dataset with one request is unreliable; the monitoring service instead runs a report when its window fills and replaces older rows as new requests arrive. section 6.2.3
Two changes to investigate
Reducing the brightness of test images produces a detectable change in brightness distribution while aspect ratio and area can remain unchanged. In a movie recommender, increasing selected ratings and comparing retrained item factors reveals a shift for an affected item. Both examples make the monitored property explicit rather than treating drift as one undifferentiated score. section 11.2.1 section 11.2.2
Treat a detection as an input to the maintenance loop: inspect the affected features and model performance, then update data collection or retrain when warranted. Monitoring makes those interventions possible before changing data goes unchecked. section 2.1.7 section 6.3
