liveBook · Manning

Concept drift

As explained in LLM Customization and Fine-Tuning

A change in the relationship between a model’s input data and the concepts or outcomes it is expected to represent.

Concept drift occurs when the relationship between inputs and correct outputs changes. A model may give outdated answers even when the questions themselves have not changed. section 9.2

When a correct answer goes stale

A support model directs “password reset” questions to an Active Directory self-service portal. After the organization replaces that portal with a cloud SSO provider, the same answers are wrong. section 9.2

Do not confuse this with data drift, where the input distribution changes, or prompt drift, where users change how they phrase requests. Vocabulary-distance detectors can flag data or prompt drift, but canary prompts are needed to catch answers made stale by concept drift. section 9.2 section 9.2.3

Use a quarterly review to decide whether conditions have changed:

  • Sample recent production questions and compare their vocabulary with the training set. section 9.6.4
  • Ask support leads what changed in products, policies, processes, or terminology. section 9.6.4
  • Assess retraining when metrics indicate degradation; do not retrain automatically on a schedule. section 9.6.4
Figure 9.3 Responding to a drift signal: a red drift score triggers a canary run before any retrain decision.

A red drift score triggers a canary run before any retraining decision. section 9.2.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines four drift forms and gives freshness checks, while rejecting scheduled retraining as the default response.
Examples
Uses an Active Directory-to-SSO migration to show unchanged password-reset questions receiving outdated answers.
What you can do after
Diagnose stale answers by reviewing recent questions, training vocabulary, and changes reported by support leads.
How it connects
Distinguishes data drift, prompt drift, and model drift, and connects concept drift to canary evaluation.

Also explained in

Compare how 8 books explain concept drift →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.