liveBook · Manning

Data drift

As explained in AI Model Evaluation

Feature drift detection is the identification of changes over time in the distributions or behavior of model input features.

Data drift is a change over time in the data or environment a model relies on. Feature drift specifically means that the distribution of model inputs has changed. A model that performed well offline may see different inputs in production and become less effective as user behavior, content, or product conditions change. section 6.4.2 section 1.7.1

Monitor important features before and during an A/B test, especially features tied to user behavior or time. Compare their production distributions with training data, and alert when a shift exceeds a chosen threshold. Continue monitoring after launch and through retraining cycles. section 6.4.2 section 6.8 section 8.7.2

Two statistical checks can help flag distribution changes:

  • The Kolmogorov–Smirnov (KS) test asks whether two samples come from the same distribution. A low p-value can trigger a drift alert. section 6.4.2
  • The Population Stability Index (PSI) measures the shift between distributions. An example classification treats PSI below 0.1 as no significant change, 0.1 to below 0.25 as a minor change, and 0.25 or higher as a major shift. section 6.4.2
Listing 6.1 KS Test Implementation to Detect Feature Distribution Drift
def kolmogorov_smirnov_test(training_feature, production_feature, feature_name, threshold=0.05):
    """
    Perform KS test to detect distribution drift
    """
    statistic, p_value = stats.ks_2samp(training_feature, production_feature)
    
    drift_detected = p_value < threshold
    
    result = {
        'feature': feature_name,
        'ks_statistic': statistic,
        'p_value': p_value,
        'drift_detected': drift_detected,
        'severity': 'HIGH' if statistic > 0.2 else 'MEDIUM' if statistic > 0.1 else 'LOW'
    }
    
    return result
 

This implementation compares training and production samples and returns a drift flag, test statistic, p-value, and severity. section 6.4.2

Listing 6.2 Calculating PSI
def population_stability_index(training_feature, production_feature, bins=10):
    """
    Calculate Population Stability Index (PSI) to measure drift
    PSI < 0.1: No significant change
    0.1 ≤ PSI < 0.25: Some minor change
    PSI ≥ 0.25: Major shift in population
    """
    # Create bins based on training data
    _, bin_edges = np.histogram(training_feature, bins=bins)
    
    # Calculate distributions
    training_counts, _ = np.histogram(training_feature, bins=bin_edges)
    production_counts, _ = np.histogram(production_feature, bins=bin_edges)
    
    # Normalize to percentages and avoid division by zero
    training_pct = training_counts / len(training_feature)
    production_pct = production_counts / len(production_feature)
    
    # Add small constant to avoid log(0)
    training_pct = np.maximum(training_pct, 0.0001)
    production_pct = np.maximum(production_pct, 0.0001)
    
    # Calculate PSI
    psi = np.sum((production_pct - training_pct) * np.log(production_pct / training_pct))
    
    return psi

This implementation starts by binning training and production feature values to calculate PSI. section 6.4.2

A drift alert is an early warning, not proof that the model is wrong or that the shift is harmful. Check performance on fresh production data, analyze relevant segments, and use domain knowledge to investigate. Some evaluation metrics require delayed labels or human review. Performance changes can also arise from a changed relationship between inputs and outcomes, rather than input drift alone. section 2.3.4 section 1.7.1

Changing job preferences

A job recommender that once performed well may face more remote-role searches or a different mix of users. Compare historical and current query patterns, job categories, location preferences, and engagement. If the model still prioritizes on-site roles while remote preferences grow, possible responses include retraining on fresher data, adding a remote-intent feature, adjusting ranking rules, or inspecting affected user segments. section 3.4.2

Track data drift alongside metric drift: input distributions can shift while online measures such as click-through rate or satisfaction also change. If offline metrics predict stability but live metrics drift, investigate the evaluation framework as well as the model. section 8.7.2

How AI Model Evaluation covers it

How deep it goes
Defines feature drift and demonstrates KS and PSI checks with alerts and example thresholds, while cautioning that shifts alone do not establish harm.
Examples
Uses changing remote-job preferences and seasonal shifts in user behavior to illustrate changing inputs.
What you can do after
Diagnose/debug with KS and PSI checks, then investigate flagged shifts using fresh performance data and segment analysis.
How it connects
Connects distribution shift to offline evaluation, A/B testing, retraining, metric drift, and segment analysis.

Also explained in

Compare how 11 books explain data drift →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.