Data drift is a change over time in the data or environment a model relies on. Feature drift specifically means that the distribution of model inputs has changed. A model that performed well offline may see different inputs in production and become less effective as user behavior, content, or product conditions change. section 6.4.2 section 1.7.1
Monitor important features before and during an A/B test, especially features tied to user behavior or time. Compare their production distributions with training data, and alert when a shift exceeds a chosen threshold. Continue monitoring after launch and through retraining cycles. section 6.4.2 section 6.8 section 8.7.2
Two statistical checks can help flag distribution changes:
- The Kolmogorov–Smirnov (KS) test asks whether two samples come from the same distribution. A low p-value can trigger a drift alert. section 6.4.2
- The Population Stability Index (PSI) measures the shift between distributions. An example classification treats PSI below 0.1 as no significant change, 0.1 to below 0.25 as a minor change, and 0.25 or higher as a major shift. section 6.4.2
Listing 6.1 KS Test Implementation to Detect Feature Distribution Drift
def kolmogorov_smirnov_test(training_feature, production_feature, feature_name, threshold=0.05):
"""
Perform KS test to detect distribution drift
"""
statistic, p_value = stats.ks_2samp(training_feature, production_feature)
drift_detected = p_value < threshold
result = {
'feature': feature_name,
'ks_statistic': statistic,
'p_value': p_value,
'drift_detected': drift_detected,
'severity': 'HIGH' if statistic > 0.2 else 'MEDIUM' if statistic > 0.1 else 'LOW'
}
return result
This implementation compares training and production samples and returns a drift flag, test statistic, p-value, and severity. section 6.4.2
Listing 6.2 Calculating PSI
def population_stability_index(training_feature, production_feature, bins=10):
"""
Calculate Population Stability Index (PSI) to measure drift
PSI < 0.1: No significant change
0.1 ≤ PSI < 0.25: Some minor change
PSI ≥ 0.25: Major shift in population
"""
# Create bins based on training data
_, bin_edges = np.histogram(training_feature, bins=bins)
# Calculate distributions
training_counts, _ = np.histogram(training_feature, bins=bin_edges)
production_counts, _ = np.histogram(production_feature, bins=bin_edges)
# Normalize to percentages and avoid division by zero
training_pct = training_counts / len(training_feature)
production_pct = production_counts / len(production_feature)
# Add small constant to avoid log(0)
training_pct = np.maximum(training_pct, 0.0001)
production_pct = np.maximum(production_pct, 0.0001)
# Calculate PSI
psi = np.sum((production_pct - training_pct) * np.log(production_pct / training_pct))
return psi
This implementation starts by binning training and production feature values to calculate PSI. section 6.4.2
A drift alert is an early warning, not proof that the model is wrong or that the shift is harmful. Check performance on fresh production data, analyze relevant segments, and use domain knowledge to investigate. Some evaluation metrics require delayed labels or human review. Performance changes can also arise from a changed relationship between inputs and outcomes, rather than input drift alone. section 2.3.4 section 1.7.1
Changing job preferences
A job recommender that once performed well may face more remote-role searches or a different mix of users. Compare historical and current query patterns, job categories, location preferences, and engagement. If the model still prioritizes on-site roles while remote preferences grow, possible responses include retraining on fresher data, adding a remote-intent feature, adjusting ranking rules, or inspecting affected user segments. section 3.4.2
Track data drift alongside metric drift: input distributions can shift while online measures such as click-through rate or satisfaction also change. If offline metrics predict stability but live metrics drift, investigate the evaluation framework as well as the model. section 8.7.2










