Backtesting compares a model’s predictions with outcomes that actually occurred in historical data. It supports offline evaluation only when the data reflects real product scenarios and user behavior; otherwise, its metrics can mislead product and engineering decisions. section 1.3.3
Build a useful backtesting setup by separating data roles and checking representativeness:
- Use historical production events, user behavior, system decisions, labels, outcomes, or interactions as evaluation inputs. section 1.3.3
- Keep training, validation, and holdout data distinct: training supports learning, validation supports tuning and iteration, and holdout data supports a more independent final assessment. section 1.3.3
- Treat synthetic data as supplemental: it can cover rare failures or missing cases, but requires validation against realistic production patterns. section 1.3.3
Recommendation evaluation
A recommendation system can be backtested by comparing its predicted recommendations with recorded user interactions. Results become unreliable when the historical data reflects outdated behavior, biased exposure from an earlier model, or incomplete logging. section 1.3.3

