liveBook · Manning

Data drift

As explained in LLM Customization and Fine-Tuning

A change over time in the characteristics or distribution of data a model encounters, potentially reducing its performance.

Data drift is a change in the distribution of inputs a model encounters over time. A model can lose accuracy even when its weights and training data have not changed: production traffic has moved away from the inputs it learned to handle. section 9.2

IT support after a platform migration

An IT support model trained largely on VPN and email questions encounters a new cloud platform. Six months later, 40% of incoming questions concern Kubernetes networking, a topic barely represented in training. The input shift is a warning that the model’s original coverage may no longer match demand. section 9.2

Data drift is distinct from concept drift, where the correct answer changes, and model drift, where model behavior changes, such as after an API provider update. Prompt drift describes changes in how users phrase requests, such as adopting conversational shorthand. section 9.2

A similar distribution mismatch can exist at launch: training and validation examples may be clean and curated while production queries contain typos, incomplete sentences, or mixed languages. Validation metrics can look healthy if the validation set shares the training set’s bias. section 6.8.3

Reduce mismatch and maintain production quality:

  • Sample training examples from real production traffic where possible. Include ambiguous, incomplete, out-of-scope, and typo-containing requests; validate on a separate held-out set drawn from production. section 6.8.3
  • Anchor synthetic examples to real seed examples so their style and vocabulary do not introduce a separate shift. Treat a 30% synthetic-data cap as a first-run heuristic, not a universal threshold. section 3.4.3
  • Monitor input distributions and periodically run canary prompts and evaluations. Calibrate drift thresholds against your own data; a detector alert is a leading indicator, not proof that retraining is needed. section 9.11 section 9.7.1
  • Investigate alerts using canary results, user feedback, and recent outputs. If meaningful degradation is confirmed, sample recent traffic, add targeted examples, retrain, and compare against the previous baseline. section 9.7.1 section 3.5.3

Schedule evaluations with current queries, for example quarterly, rather than waiting for users to notice degradation. Retraining carries ongoing costs; use simpler approaches when they meet the quality bar, and gate retrained versions through the evaluation suite before deployment. section 5.10 section 6.13.1 section 9.11

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines input drift, distinguishes related drift types, and gives monitoring and retraining decision steps.
Examples
Illustrates drift through IT support questions shifting toward Kubernetes networking after a cloud migration.
What you can do after
Diagnose input drift with a TF-IDF cosine-centroid detector and canary prompts, then investigate whether output degradation warrants retraining.
How it connects
Connects data drift to covariate shift, concept drift, prompt drift, model drift, and distribution mismatch.

Also explained in

Compare how 11 books explain data drift →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.