liveBook · Manning

Online evaluation

As explained in AI Model Evaluation

Evaluation of an AI system in a live or production environment using real-time system behavior or user interactions.

Online evaluation measures how an AI system behaves with real users in a live product. It tests whether a change improves the user experience under actual product, latency, and behavioral conditions—not just whether it looks promising against historical data. section 1.1 section 6.0

Offline and online evaluations are complementary. Offline tests can screen out weak models, uncover safety problems, and predict potential impact before users are exposed. They cannot establish real-world impact: a small offline gain may not produce meaningful engagement or conversion gains. Compare offline and online outcomes over time to understand and improve their alignment. section 1.6 section 2.3.1

A ranking improvement with a downstream cost

A ranking model may perform well against historical labels or raise click-through rate in production, yet leave users with shorter sessions, lower satisfaction, or higher latency. Evaluate the product experience beyond the immediate metric before deciding whether to launch. section 6.0

A common online method is an A/B test: use a controlled experiment to learn how users respond to a model change and whether the benefit justifies its cost and complexity. The aim is to inform a product decision, not to confirm that an appealing model must be better. section 6.0 section 6.1.1

Before exposing a candidate model to users, establish offline exit criteria appropriate to the product:

  • Beat the current baseline on the primary offline metric while keeping guardrails such as latency, cost, safety, fairness, or calibration within acceptable ranges. section 6.2
  • Check important user and content segments for unacceptable regressions, and understand remaining errors well enough to monitor them online. section 6.2
  • Test on realistic, recent, representative data. State what the online experiment should improve and which trade-offs are acceptable. section 6.2

Meeting these criteria makes a model a candidate for online testing; it does not guarantee success. Timing also depends on model maturity, delivery expectations, and confidence in the offline evaluation. Early models may benefit from lightweight directional tests, while mature models call for more rigorous evaluation. section 6.2 section 6.8

Interpret online results carefully. More model variants increase experiment complexity and can require larger samples or longer runs. Cold-start and novelty effects can distort early results; feature drift can degrade performance during a test. Watch for weak proxy metrics, feedback loops, segment blind spots, and noisy effects rather than relying on a single metric or p-value. section 6.8 section 8.0

How AI Model Evaluation covers it

How deep it goes
Explains readiness criteria, A/B testing trade-offs, and interpretation pitfalls while emphasizing that offline results cannot establish live impact.
Examples
Uses ranking models, click-through rates, session length, satisfaction, and latency to illustrate product trade-offs.
What you can do after
Evaluate and understand whether a model is ready for live testing and how its online results inform a product decision.
How it connects
Connects offline evaluation, A/B testing, guardrail metrics, feature drift, and feedback loops.

Also explained in

Compare how 3 books explain online evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.