Data representativeness means making training, validation, and test data resemble the cases a model will encounter in practice. A convincing score depends on that match. section 5.2.3 section 6.2.2
The ordered-digit trap
Suppose digit samples are ordered by class. Taking the first 80% for training and the last 20% for testing puts classes 0–7 in training and classes 8–9 in testing. Randomly shuffling before splitting usually prevents this mismatch. section 5.2.3
Use these practical checks when choosing an evaluation protocol:
- Shuffle before splitting when observations have no meaningful time order. section 5.2.3
- Preserve chronological order when predicting the future from the past; otherwise, future data leaks into training. section 5.2.3
- Keep training and validation sets disjoint, especially when duplicate samples exist. section 5.2.3
- Choose hold-out validation, K-fold cross-validation, or iterated K-fold validation according to dataset size. section 6.2.2
