A dataset split partitions observations so a model can be trained and evaluated without treating the same data as both practice and a final exam. A common arrangement uses a training set to fit the model, a validation set to compare models or tune hyperparameters, and a test set reserved for final metrics. That arrangement is reliable only if the evaluation data represents the conditions the model will face later. section 7.1 section 7.2.1
Choose the split around the decision you need to make:
- Set proportions according to sample count and the precision and reliability required of the metrics; 60/20/20 is an example, not a rule. section 7.2.1
- Keep the final test set out of model selection. Repeatedly optimizing against one validation split can overfit it too. section 7.1 section 7.2.1
- Check whether the split resembles production and revisit validation when data sources, features, or feedback from model use change. section 7.1
Keep a search user's queries together
For an image-search system, validation and test queries should reflect real users and cover varied topics. Group queries by user, remove duplicates, then assign each user to exactly one split. A possible starting ratio is 90/5/5, but the best ratio is not known in advance. Assign new users consistently rather than reshuffling them: otherwise, queries from one user could cross sets and leak information. section 7.5.2
def assign_bucket(user_id):
_hash = sha1(user_id.encode()).hexdigest()
return int(_hash, 16) % n_buckets
def assign_split(user_id):
bucket = assign_bucket(user_id)
if bucket < n_buckets * train_ratio:
return 'train'
elif bucket < n_buckets * (train_ratio + val_ratio):
return 'val'
else:
return 'test'
This Python listing hashes a user ID into a bucket and maps buckets to train, validation, or test. Stable user-level assignment also lets the allocation change by moving buckets. section 7.5.2
Random assignment is not right for every system. In a retail forecasting pipeline, training and validation data should not overlap in time, to prevent leakage. If a learning curve shows a persistent gap between training and validation performance, inspect the split as well as the model: the two sets may represent different conditions. section 10.6.1 section 9.1.5
Splitting also matters when comparing systems in an experiment. Choose the assignment unit before launch: users may work, but stores or geographic areas may be necessary. Larger units can produce unequal or unrepresentative groups, so check their similarity and choose the statistical test accordingly. A switchback experiment instead alternates control and test assignments across region-time buckets. section 12.2.1 section 12.2.2
