A deployed model can degrade even when its weights never change: production inputs shift, base models are superseded, and safety boundaries face new adversarial techniques. Drift detection surfaces changes before they become incidents, but a change in inputs alone does not prove that retraining is necessary. section 9.0 section 9.9.1
No single detector covers data, concept, prompt, and model drift. A practical combination uses a cheap distribution-level detector continuously and reruns fixed canary prompts periodically to check specific model behaviors against a saved baseline. section 9.2.1 section 9.0
Table 9.3 Drift detection methods
|
Method
|
Detects
|
Cost
|
When to use
|
|
TF-IDF/embedding cosine distance
|
Data drift, prompt drift
|
Low (no GPU)
|
Always, cheap baseline
|
|
PSI (Population Stability Index)
|
Distribution shift in features
|
Low
|
Structured input features
|
|
KL divergence on token
distributions
|
Output distribution change
|
Medium
|
When you log model outputs
|
|
Canary prompts
|
Model drift, concept drift
|
Medium
|
Monthly schedule
|
|
LLM-as-judge comparison
|
Quality degradation
|
High
|
Quarterly deep evaluation
|
Compare detection methods by the changes they detect, their cost, and when to use them. section 9.2.1
A lightweight text detector builds a shared TF-IDF vocabulary from reference and production texts, averages each set’s vectors into a centroid, and computes drift as 1 minus their cosine similarity. A score of zero means the centroids are identical; one means they are orthogonal. TF-IDF is inexpensive but treats text as a bag of words and ignores order and grammar. Sentence-transformer embeddings can replace the vectorizer while retaining the centroid comparison. section 9.2.2
Listing 9.2 drift_detector.py (key sections)
# (key sections -- full source in code/chapter09/drift_detector.py)
# Alert thresholds #A
DRIFT_GREEN = 0.10 # No action
DRIFT_YELLOW = 0.20 # Investigate
# Above 0.20 = RED (consider retraining)
def alert_level(drift_score: float) -> str: #B
if drift_score < DRIFT_GREEN:
return "green"
if drift_score < DRIFT_YELLOW:
return "yellow"
return "red"
def compute_drift(
reference_texts: list[str],
production_texts: list[str],
) -> dict:
"""Compute distribution-level drift between two text sets.
Builds a shared TF-IDF vocabulary, then measures centroid
similarity between the reference and production distributions.
"""
all_texts = reference_texts + production_texts
n_ref = len(reference_texts)
matrix, vocab = build_tfidf_matrix(all_texts)
ref_matrix = matrix[:n_ref]
prod_matrix = matrix[n_ref:]
# Centroid similarity
ref_centroid = ref_matrix.mean(axis=0)
prod_centroid = prod_matrix.mean(axis=0)
centroid_sim = cosine_similarity(ref_centroid, prod_centroid)
drift_score = 1.0 - centroid_sim #C
# Per-prompt similarity to reference centroid
per_prompt_sims = []
for i in range(prod_matrix.shape[0]):
sim = cosine_similarity(prod_matrix[i], ref_centroid)
per_prompt_sims.append(float(sim))
# ...
return { #D
"centroid_similarity": round(centroid_sim, 4),
"drift_score": round(drift_score, 4),
"alert_level": alert_level(drift_score),
"num_reference": n_ref,
"num_production": len(production_texts),
"vocab_size": len(vocab),
}Inspect the NumPy-based detector and its starting alert thresholds. section 9.2.2
The example thresholds are green below 0.10, yellow from 0.10 to below 0.20, and red at 0.20 or above. Investigate yellow and consider retraining at red, but calibrate thresholds to your own data: running the detector on a train/validation split establishes a baseline. section 9.2.2 section 9.11
Wait for corroborating signals
In an illustrative support deployment, a ticketing-system migration produces a yellow drift score of 0.18, but canary Token-F1 remains 0.91. The team logs the alert and keeps monitoring. Later, two consecutive red weeks, canary F1 of 0.74, and complaints about outdated instructions justify preparing new data and retraining. Acting on the first signal alone would have spent engineering time and GPU budget without evidence of broken behavior. section 9.9.1 section 9.0
Assign an owner to a weekly drift review and publish the maintenance cadence. When evidence warrants retraining, gate the updated model through the full evaluation suite before deployment. Keep rollback ready as well: detection identifies change, but rollback determines how quickly a team can respond to a failing release. section 9.6 section 9.11 section 9.3
