A demand forecast can look accurate offline yet disappoint in production if its evaluation uses data that would not have arrived when the forecast was made. Validation structures training and evaluation to imitate the knowledge available during real operation. Its primary job is to estimate predictive power on unseen data reliably enough to guide model and feature choices. section 7.5.1 section 7.0
Choose a procedure by checking what separates past examples from future use:
- Start with distinct training, validation, and test sets: train the model on the first, evaluate during development on the second, and calculate final metrics on the third. Check the assumption that these sets and future data have comparable distributions. section 7.1
- Keep related examples from leaking across splits. For time-sensitive data, put evaluation after training in time rather than randomly mixing past and future. section 7.2.4 section 7.3
- Match the evaluation window to the forecast horizon and retraining schedule. Consider training-history length, seasonality, and a gap when new data or features arrive late. section 7.2.4
- Do not repeatedly search for the best model against one validation split and treat its score as unbiased. Separate tuning from final estimation, and revisit the procedure when data sources, features, or feedback loops change. section 7.1 section 7.5.1
Forecasting retail demand
For a four-week forecast, use rolling evaluation windows of 28 days and train each model on the preceding two years of available data. Leave a three-day gap before each evaluation window to reflect delayed arrivals; the gap also constrains which features can be calculated. Use an inner loop for hyperparameter optimization and an outer loop for performance estimation. section 7.5.1
Evaluating search queries
For search, make the user—not the individual query—the unit of split assignment. Keep one user's queries in a single split and remove duplicate queries to reduce leakage. Aim for validation and test queries that reflect real users and cover diverse topics; check rather than assume that random assignment handles seasonal or changing query patterns. section 7.5.2
def assign_bucket(user_id):
_hash = sha1(user_id.encode()).hexdigest()
return int(_hash, 16) % n_buckets
def assign_split(user_id):
bucket = assign_bucket(user_id)
if bucket < n_buckets * train_ratio:
return 'train'
elif bucket < n_buckets * (train_ratio + val_ratio):
return 'val'
else:
return 'test'
Hash-based buckets illustrate how to assign each user to a stable split while allowing split ratios to change by reassigning buckets. section 7.5.2
A standard schema is a starting point, not a guarantee. If ordinary or combined splits fail to represent the difference between seen and unseen data, design a procedure for that difference. Weigh bias, variance, and computation time when choosing folds, and use different schemas together when they serve different evaluation goals. section 7.3 section 7.6
