A probability distribution describes how values or observations are spread across a dataset or population. In machine learning, it matters because collected data may not represent the real distribution encountered after deployment. section 6.5
Use distribution checks to ask practical questions:
- Do training, validation, and test datasets come from the same distribution, and will that remain true in the future? section 7.1
- Can a model distinguish labeled from unlabeled data, revealing which features best predict a distribution difference? section 7.3.2
- Have changing data distributions made features less relevant, fragile, or difficult to reproduce? section 11.1.1
- Are shifts in specific feature distributions affecting downstream stages or signaling a corrupted input pipeline? section 14.1.1
Adversarial validation
Train a classifier to separate labeled and unlabeled datasets. If it separates them effectively—for example, with ROC AUC above 0.6—use its output as a measure of dataset proximity, then inspect the features driving the difference. section 7.3.2
Treat distribution assumptions as hypotheses rather than guarantees. Recheck them when new data sources, features, feedback loops, or production conditions appear, because repeated validation is more reliable than a one-time split. section 7.1 section 6.5
