liveBook · Manning

Drift

As explained in AI Model Evaluation

A change over time in a model's data distribution, behavior, or performance relative to its earlier state.

Model drift is a change over time in a model or evaluation judge that can undermine evaluation while outputs remain plausible. A hosted judge may change behavior behind a stable-looking API identifier even when its prompt, rubric, and pipeline stay unchanged; score changes may then reflect the judge, not the product. section 10.6.1

Separate the possible causes and use evaluations suited to the setting:

  • For hosted judges, use canary evaluation to help distinguish a drifting judge from a changing product. section 10.6.1
  • For internal models, offline evaluation can track performance over time. Historical support tickets can be compared with agent labels using accuracy, critical-ticket recall, or resolution-time proxies. section 1.3.2
  • For internal support agents, replay historical tickets and assess outputs against known resolutions or human-written responses. section 1.3.2

How AI Model Evaluation covers it

How deep it goes
Distinguishes judge drift from product changes and points to canary or offline evaluation, without detailing a full monitoring procedure.
Examples
Uses historical support tickets, agent labels, and replayed outputs, alongside hosted-judge behavior changes behind stable API identifiers.
What you can do after
Diagnose whether evaluation score changes reflect product behavior or a drifting hosted judge, and assess internal models offline over time.
How it connects
Links model drift to offline evaluation, canary evaluation, and Goodhart's law.

Also explained in

Compare how 18 books explain drift →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.