A dataset split separates observations into a train subset for fitting a model and a test subset for evaluating predictions on unseen data. The comparison helps distinguish learned signal from noise and reveals overfitting: a model that memorizes past data may perform poorly on future cases. section 4.1
Studying for an exam
Instead of studying every solved exam and risking memorization, set some exams aside, study the rest, and test yourself on the untouched subset. Likewise, fit the model on train data, predict the test outcomes, and compare predictions with what actually happened. The resulting error estimates future performance on new production data. section 4.1
Use the split as an evaluation safeguard, while keeping its limits in view:
- Select the final model from candidate functions and configurations by minimizing error on the test set. section 4.1
- Treat train-test splitting as a form of cross-validation, but do not assume cross-validation prevents every kind of overfitting. section 4.1 section 5.3.2
- For propensity scores, use cross-fitting when predictions are calculated for the same historical patients used to construct the predictive models. section 5.3.2
- Do not choose causal-inference variables solely because they improve predictive accuracy; domain knowledge should guide adjustment choices. section 5.3.2
