Delayed shift validation keeps a short validation period fixed while moving it at infrequent intervals, reducing metric noise caused by constantly changing data. section 7.2.4
Use it when daily validation changes obscure whether model quality has actually moved:
- Keep the validation window one week long. section 7.2.4
- Shift the window once a month instead of daily. section 7.2.4
- Compare models against the same fixed validation data throughout each month. section 7.2.4
Changing product mix
When product variety changed 15% week to week and 40% month to month, daily validation updates made metrics fluctuate even without genuine model-quality changes. Monthly shifts produced fresher, substantially less noisy comparisons. section 7.2.4
Figure 7.4 Standard time-based split. The test dataset always follows the train one, so train samples are “past” and test is “future.”
Use the standard time-based split as a reference for keeping training in the past and validation in the future. section 7.2.4
