A long-term holdback reserves part of production traffic on an invariant baseline model. It anchors online evaluation, separating genuine improvement from feedback-driven distribution shift. section 8.2.1 section 8.7.4
Design the holdback around the product’s cost and the evaluation question:
- Keep the holdback continuously when possible; this provides the most reliable anchor for detecting slow feedback-driven drift. section 8.2.1
- Use quarterly runs or post-retraining and product milestones when continuous maintenance or user exposure to an old model is too costly. section 8.2.1
- Rotate cohorts during periodic holdbacks to distribute impact while preserving comparison across evaluation periods. section 8.2.1
- Maintain the model versions, datasets, and pipelines needed to keep the baseline stable and the comparison interpretable. section 8.2.1
Movie recommendation baseline
Keep 2% of users on the old recommender indefinitely while the remaining users receive the new model. Compare watch time and catalog diversity against this invariant group to distinguish real gains from feedback-induced drift. section 8.7.4
Figure 8.1 illustrates feedback loop in AI model evaluations. The model influences user behavior, which generates logged data that feeds back into retraining. Pitfalls such as exposure bias, reinforced errors, and satisfaction masking emerge along the loop, making evaluation an evolving target rather than a clean snapshot.
Trace how model recommendations change user behavior, alter logged data, and influence retraining; the loop highlights exposure bias, reinforced errors, and satisfaction masking. section 8.2
