An evaluation dataset is data set aside to measure a model rather than train it. In a common three-way split, the training set fits the model, the validation set compares models and tunes hyperparameters during development, and the test set provides final metrics. For example, a 60/20/20 split is possible, but the right proportions depend on sample count and how reliable the metrics need to be. section 7.1 section 7.2.1
A clean split is not automatically a reliable one. It assumes the partitions represent the same distribution and that this distribution remains relevant later. Repeatedly choosing features, models, or hyperparameters against one validation set can overfit decisions to that set; using the test set for those decisions compromises the final check too. Revisit the evaluation scheme when new data sources, features, or feedback from model use change the system. section 7.1
Choose how the evaluation data changes with the problem:
- Use a moving recent window when old observations no longer reflect the target distribution. One option is to update a two-week validation window daily. section 7.4
- Preserve the training-to-validation ratio as newly labeled data arrives when recency is less important, rather than sending all new examples to just one partition. section 7.4
- Keep an unchanged golden set when the priority is comparing models against the same benchmark over a long period. It may be sampled or assembled from diverse hard cases. section 7.4
Two practical split decisions
For a system updated with weekly data and labels, update its validation split weekly to catch local performance changes; a separate benchmark can track longer-term improvement. For a search system, split by user rather than by query, remove duplicate queries, and keep each user's queries in one partition. This helps prevent leakage while making validation and test queries representative of real use. section 7.5.1 section 7.5.2
def assign_bucket(user_id):
_hash = sha1(user_id.encode()).hexdigest()
return int(_hash, 16) % n_buckets
def assign_split(user_id):
bucket = assign_bucket(user_id)
if bucket < n_buckets * train_ratio:
return 'train'
elif bucket < n_buckets * (train_ratio + val_ratio):
return 'val'
else:
return 'test'
The user-ID bucketing example shows how to assign users consistently to training, validation, or test partitions; changing bucket allocations can change their proportions. section 7.5.2
