An evaluation dataset is kept separate from model fitting so performance can be measured on observations the model has not seen. A held-out test set provides this check; cross-validation repeats it across training and validation folds. For rare outcomes such as fraud, stratifying the split preserves similar class proportions and helps prevent an unrepresentative evaluation. section 5.3.5 section 8.2.2
Practical checks:
- In fraud detection, a 70/30 stratified split keeps the fraud proportion near 0.1725% in both sets. section 8.2.2
- For credit scoring, five-fold validation can provide more stable metrics; retain a hold-out set to guard against overfitting. section 5.3.5
