liveBook · Manning

Sandboxing

As explained in AI Model Evaluation

An isolated execution environment that enables experimentation without affecting real users, systems, or production traffic.

Sandboxing isolates experimentation from real users, production systems, and live traffic. It provides a controlled space to compare model versions, inspect behavior, and iterate without disrupting the user experience. section 2.1

A sandbox supports several practical evaluation activities:

  • Run models against batch-processed historical interactions, labeled datasets, or simulated online context. section 2.1
  • Compare model versions and measure how changes affect selected metrics before product iteration. section 2.1
  • Investigate feature drift, unintended bias, and behavior under different conditions without exposing users to the change. section 2.1
Figure 2.1 visualizes how offline evaluations serve as the central concept, with three key categories—performance evaluations, diagnostic evaluations, and simulation-based evaluations—contributing independently. These categories feed into combined evaluations, which integrate insights from all three to provide a holistic view of model behavior.

Map sandboxed offline evaluation across performance, diagnostic, simulation-based, and combined evaluations. section 2.1

How AI Model Evaluation covers it

How deep it goes
Defines isolation and outlines historical-data testing, metric analysis, drift diagnosis, and bias investigation without implementation internals.
Examples
Uses historical interaction logs, labeled datasets, on-demand model invocations, model-version comparisons, and metric analysis.
What it assumes
Assumes model versions, historical data, metrics, and production environments; introduces sandboxing through offline evaluation.
What you can do after
Evaluate/understand model changes, investigate failures, and explore product optimizations without risking real users or systems.
How it connects
Connects offline evaluation with performance evaluations, diagnostic evaluations, simulation-based evaluations, feature drift, and unintended bias.

Also explained in

Compare how 10 books explain sandboxing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.