liveBook · Manning

Population stability index

As explained in AI Model Evaluation

The Population Stability Index measures the degree of distribution shift between two datasets or populations.

Population Stability Index (PSI) measures how much a distribution shifts between two populations, such as training data and production traffic. It helps explain why offline evaluation may fail to generalize when sampled data no longer represents real usage. section 2.3.4

Use PSI as an early-warning diagnostic, not as a standalone verdict:

  • Compare feature distributions between training and production data. section 6.4.2
  • Interpret PSI below 0.1 as no significant change, 0.1–0.25 as minor change, and 0.25 or higher as a major population shift. section 6.4.2
  • Investigate high values with performance checks, segment analysis, and domain knowledge because PSI does not show whether the shift is harmful or whether labels and performance changed. section 2.3.4
  • Use alerts to decide whether to pause an A/B test, investigate a root cause, or adjust analysis for population changes. section 6.4.2

Training data versus production traffic

A monitoring job creates bins from the training feature, counts training and production observations in those bins, normalizes the counts, prevents zero proportions, and calculates PSI. A rising PSI beside retention or other live metrics signals that evaluation reliability deserves investigation rather than proving model failure. section 6.4.2 section 8.7.2

Listing 6.2 Calculating PSI
def population_stability_index(training_feature, production_feature, bins=10):
    """
    Calculate Population Stability Index (PSI) to measure drift
    PSI < 0.1: No significant change
    0.1 ≤ PSI < 0.25: Some minor change
    PSI ≥ 0.25: Major shift in population
    """
    # Create bins based on training data
    _, bin_edges = np.histogram(training_feature, bins=bins)
    
    # Calculate distributions
    training_counts, _ = np.histogram(training_feature, bins=bin_edges)
    production_counts, _ = np.histogram(production_feature, bins=bin_edges)
    
    # Normalize to percentages and avoid division by zero
    training_pct = training_counts / len(training_feature)
    production_pct = production_counts / len(production_feature)
    
    # Add small constant to avoid log(0)
    training_pct = np.maximum(training_pct, 0.0001)
    production_pct = np.maximum(production_pct, 0.0001)
    
    # Calculate PSI
    psi = np.sum((production_pct - training_pct) * np.log(production_pct / training_pct))
    
    return psi

Inspect the supplied Python listing for a thresholded PSI calculation using training-derived bins and production distributions. section 6.4.2

Figure 8.5 illustrates a metric drift dashboard mock-up. The left panel shows CTR over time where the stable series fluctuates within normal variance, while the drifting series trends upward beyond the baseline, suggesting feedback-driven distortion. The right panel tracks retention and PSI (Population Stability Index).

Use the dashboard mock-up to relate PSI to retention and distinguish ordinary metric movement from input drift that can undermine evaluation. section 8.7.2

How AI Model Evaluation covers it

How deep it goes
Explains thresholds, binning, alerts, and interpretation limits, but stops short of proving causality or model degradation.
Examples
Uses training-versus-production features, histogram bins, drift alerts, and dashboards combining PSI with CTR or retention.
What it assumes
Assumes comparable datasets and basic distribution monitoring, while teaching threshold interpretation and diagnostic limitations.
What you can do after
Diagnose distribution drift with PSI, then validate its product impact through performance checks, segments, and domain knowledge.
How it connects
Connects distribution shift with feature drift, A/B testing, offline evaluation, performance checks, and Kolmogorov–Smirnov testing.

Also explained in

Compare how 3 books explain population stability index →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.