liveBook · Manning

Data drift

As explained in Timeless Algorithms, The Foundational Ideas

A distribution shift is a change in the input data distribution between the conditions under which a model was trained and those under which it is evaluated or applied.

A pedestrian detector trained mostly on clear daytime images may fail at night or in severe weather. Data drift occurs when statistical patterns in production differ from those represented during training, weakening the relationships a model relies on. section 11.6.3

Shifts can arise when production changes:

  • Populations, environments, sensors, or behavior differ from the training conditions. section 11.6.3
  • A model may rely on correlations—such as background or lighting—that stop predicting well under new conditions. section 11.6.3

Strong validation and test results offer limited assurance when those data resemble the training source. Production performance therefore needs monitoring as operating conditions change. Shifts can reduce accuracy, while small changes to an individual input can also sharply alter a prediction. section 11.6.3 section 11.6.4

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines the train-to-production mismatch and traces causes and failure modes, but does not prescribe a detection procedure.
Examples
Uses a pedestrian detector trained on clear daytime images to illustrate failures at night, in severe weather, or with obstructions.
What you can do after
Diagnose production risk by monitoring model performance as operating conditions change.
How it connects
Connects distribution shift to representativeness, validation and test data, calibration, and model reliability.

Also explained in

Compare how 11 books explain data drift →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.