Sandboxing isolates experimentation from real users, production systems, and live traffic. It provides a controlled space to compare model versions, inspect behavior, and iterate without disrupting the user experience. section 2.1
A sandbox supports several practical evaluation activities:
- Run models against batch-processed historical interactions, labeled datasets, or simulated online context. section 2.1
- Compare model versions and measure how changes affect selected metrics before product iteration. section 2.1
- Investigate feature drift, unintended bias, and behavior under different conditions without exposing users to the change. section 2.1
Figure 2.1 visualizes how offline evaluations serve as the central concept, with three key categories—performance evaluations, diagnostic evaluations, and simulation-based evaluations—contributing independently. These categories feed into combined evaluations, which integrate insights from all three to provide a holistic view of model behavior.
Map sandboxed offline evaluation across performance, diagnostic, simulation-based, and combined evaluations. section 2.1









