Online evaluation measures how an AI system behaves with real users in a live product. It tests whether a change improves the user experience under actual product, latency, and behavioral conditions—not just whether it looks promising against historical data. section 1.1 section 6.0
Offline and online evaluations are complementary. Offline tests can screen out weak models, uncover safety problems, and predict potential impact before users are exposed. They cannot establish real-world impact: a small offline gain may not produce meaningful engagement or conversion gains. Compare offline and online outcomes over time to understand and improve their alignment. section 1.6 section 2.3.1
A ranking improvement with a downstream cost
A ranking model may perform well against historical labels or raise click-through rate in production, yet leave users with shorter sessions, lower satisfaction, or higher latency. Evaluate the product experience beyond the immediate metric before deciding whether to launch. section 6.0
A common online method is an A/B test: use a controlled experiment to learn how users respond to a model change and whether the benefit justifies its cost and complexity. The aim is to inform a product decision, not to confirm that an appealing model must be better. section 6.0 section 6.1.1
Before exposing a candidate model to users, establish offline exit criteria appropriate to the product:
- Beat the current baseline on the primary offline metric while keeping guardrails such as latency, cost, safety, fairness, or calibration within acceptable ranges. section 6.2
- Check important user and content segments for unacceptable regressions, and understand remaining errors well enough to monitor them online. section 6.2
- Test on realistic, recent, representative data. State what the online experiment should improve and which trade-offs are acceptable. section 6.2
Meeting these criteria makes a model a candidate for online testing; it does not guarantee success. Timing also depends on model maturity, delivery expectations, and confidence in the offline evaluation. Early models may benefit from lightweight directional tests, while mature models call for more rigorous evaluation. section 6.2 section 6.8
Interpret online results carefully. More model variants increase experiment complexity and can require larger samples or longer runs. Cold-start and novelty effects can distort early results; feature drift can degrade performance during a test. Watch for weak proxy metrics, feedback loops, segment blind spots, and noisy effects rather than relying on a single metric or p-value. section 6.8 section 8.0


