Fixed shift updates validation for a changing dataset by replacing the validation window with a fixed-length set of the most recent data. It suits situations where time and novelty make older outcomes poor indicators of current performance. section 7.4
A rolling two-week validation window
Take the last two weeks, beginning with the last completed day, as validation data. Update that window daily and retrain the model used for evaluation each time. section 7.4
Choose fixed shift when:
- Recent patterns matter more than historical behavior because the target distribution changes substantially over time. section 7.4
- Evaluation must remain representative of newly arriving patterns while tracking how model updates affect performance. section 7.4
Figure 7.7 Common options for updating train/validation sets. The light data blocks are used for training, while the dark ones are used for validation.
Compare fixed shift with fixed-ratio and fixed-set strategies for updating training and validation data. section 7.4
