liveBook · Manning

Drift

As explained in LLMs in Production

Data drift is a change over time in the statistical properties or distribution of data presented to a machine-learning system.

Data drift is a change in incoming data that can make an ML model perform worse while it keeps running. For example, fake reviews can distort a recommendation system over time. Monitoring helps teams detect poor performance and adjust the model or retrain it. section 3.4.6

Text data makes drift especially difficult to assess: language representations are approximations, and correctness and data cleanliness can be hard to define. section 3.2.5

Monitor data entering and leaving the model, and consider signals such as:

  • Missing values, uniqueness, and shifts in standard deviation. section 3.4.6
  • Differences by segment, such as region or A/B test group. section 3.4.6

How LLMs in Production covers it

How deep it goes
Lists missing values, uniqueness, and standard-deviation shifts as signals, without detailing a drift-detection algorithm.
Examples
Uses fake reviews manipulating recommendations and highlights the difficulty of assessing drift in text data.
What you can do after
Diagnose silent model degradation by monitoring data and respond by adjusting the model or retraining it.
How it connects
Links data drift to monitoring, data pipelines, recommendation systems, and retraining.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 12 books explain drift →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.