Online experimentation evaluates a model or product change in a live setting, where real users interact with it and their behavior can be measured. An A/B test compares a change against a control to inform a product decision, not merely to produce a test result. section 1.3.1 section 6.0
Offline evaluation is a first filter, not a substitute for a live test. It uses pre-collected data without influencing users, so it cannot fully predict behavioral impact or the way a feature fits into a user experience. Compare offline and online outcomes over time to understand how well offline metrics predict product results. section 1.2 section 1.3.1 section 2.3.1
Before exposing a model to users, check offline exit criteria suited to the product:
- Beat the current baseline on the primary offline metric while keeping guardrails such as latency, cost, safety, fairness, and calibration within acceptable ranges. section 6.2
- Check important segments for unacceptable regressions, understand remaining errors well enough to monitor them, and test on recent, representative data. section 6.2
- State what should improve online and which trade-offs are acceptable. Passing offline checks makes a model a candidate for testing; it does not prove online success. section 6.2
Design the test around the user or product behavior the model should influence. Choose the variants, metrics, and assignment logic, then plan execution details such as sample size, duration, and rollout timing. For a new model, the control might be a heuristic; for an update, it might be the model already in production. section 6.3
Figure 6.2 A/B testing design layers, organized from outer to inner circles. The closer to the core, the more foundational and irreversible the decisions tend to be: the outer execution layer covers operational factors like test duration, rollout timing, and sample size; the middle experimental design layer defines how the test is structured, including variants, metrics, and assignment logic; and the inner core layer captures the underlying purpose of the model and the user or product behavior it’s intended to influence.
The design layers connect the test’s purpose to its experimental structure and execution. section 6.3
Recommendation explanations
A streaming app could use an LLM to explain why a title is recommended. First, evaluate explanations against historical data for relevance, grounding in user behavior, and consistency. A subsequent online experiment can measure how users actually respond—something static data cannot fully capture. section 1.3.1
Interpret results with care. Model-driven feedback loops, shifting behavior, and high-variance metrics can make signals noisy. If recommendations shape future engagement, design online tests to measure longer-term effects; for features tightly integrated with user workflows, complement testing with user studies. section 6.0 section 1.8.1 section 1.8.3
A dedicated A/B testing platform is helpful but not essential to start. Route traffic, measure key metrics, and track users over time. Without automatic bucketing, holdouts, exposure logging, and sanity checks, expect more manual work and greater risk of misleading conclusions. Use online results as a go/no-go signal for rollout. section 6.7.2 section 1.2
