liveBook · Manning

Drift

As explained in LLM Customization and Fine-Tuning

A change over time in inputs, their relationship to correct outputs, or model behavior that can degrade performance.

A production IT support assistant can lose accuracy even when its weights have not changed. New platforms, revised procedures, and changing user language can make its answers less useful or wrong. Drift is change in inputs, correct answers, phrasing, or model behavior that can erode performance over time. section 9.0 section 9.2

Distinguish four forms of drift before deciding how to respond:

  • Data drift: the input distribution changes. Questions about Kubernetes networking may grow from a rarely represented topic to 40% of requests. section 9.2
  • Concept drift: the correct response changes for the same input. A password reset request may now require an SSO provider rather than an Active Directory portal. section 9.2
  • Prompt drift: users change how they phrase requests, moving from formal questions to shorthand such as “vpn broken pls help.” section 9.2
  • Model drift: model behavior changes, as when an API provider updates the underlying model. Changed evaluation data or a retrained version can also reveal regressions. section 9.2

Monitor both input distributions and behavior. A TF-IDF cosine-centroid detector can flag distribution-level changes, while periodic canary prompts check for behavioral degradation. Calibrate thresholds on your own data; use the train/validation split to establish a baseline rather than treating default values as final. section 9.11

Make response part of the deployment lifecycle. Retrain when outcomes warrant it, starting from the most recent checkpoint, and run the full evaluation suite before deployment. Keep a registry of model and data versions and the previous two model versions ready for rollback. Assign monitoring responsibilities: evaluation sets, drift flags, and gates require ongoing engineer and reviewer time. section 9.11 section 1.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Distinguishes four drift forms and outlines detection, threshold calibration, retraining gates, and rollback procedures.
Examples
Uses an IT support assistant facing Kubernetes questions, an SSO migration, and informal user shorthand.
What you can do after
Evaluate drift by distinguishing changes in inputs, correct answers, phrasing, and model behavior.
How it connects
Connects drift to TF-IDF, cosine similarity, canary prompts, evaluation, model versioning, and rollback.

Also explained in

Compare how 18 books explain drift →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.