liveBook · Manning

Online evaluation

Online evaluation is the assessment of a system using behavior, interactions, or outcomes observed during live operation.

How 3 Manning books explain it

AI Model Evaluation

Evaluation of an AI system in a live or production environment using real-time system behavior or user interactions.

How it is used
Tests whether AI model changes improve real user experiences in production.
How deep it goes
Explains readiness criteria, A/B testing trade-offs, and interpretation pitfalls while emphasizing that offline results cannot establish live impact.
Examples
Uses ranking models, click-through rates, session length, satisfaction, and latency to illustrate product trade-offs.
What you can do after
Evaluate and understand whether a model is ready for live testing and how its online results inform a product decision.

Read how AI Model Evaluation explains online evaluation →

Designing AI Systems

Online evaluation assesses an AI system or change using live production interactions and their outcomes.

How it is used
Monitors deployed AI quality continuously by scoring sampled production traces and feeding results into improvement workflows.
How deep it goes
Defines asynchronous sampling, scorer configuration, trace attachment, dashboards, queries, and alerts, but not scorer implementation internals.
Examples
Uses a sampled patient-intake workflow with helpfulness, required-elements, retrieval-relevance, and threshold-based alert scorers.
What you can do after
Operate/configure it through scoring rules that sample live traffic, attach quality signals, expose trends, and alert on degradation.

Read how Designing AI Systems explains online evaluation →

LLM Evaluation and Alignment, The Foundational Ideas

Online evaluation is the assessment of an AI system’s outputs during production serving without blocking user requests or gating responses.

How it is used
Monitors response quality during production serving without affecting user requests.
How deep it goes
Explains execution constraints, judge-output choices, missing signals, and monitoring practices, but defers detailed judge implementation.
Examples
Uses production interaction traces and binary helpfulness or topicality judgments as monitoring artifacts.
What you can do after
Operate a non-blocking monitoring pipeline using low-cost single-answer judgments and ongoing calibration checks.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains online evaluation →

Also covered in

What to know first

Related concepts

Questions about online evaluation

What is online evaluation?

Online evaluation is the assessment of a system using behavior, interactions, or outcomes observed during live operation.

Which Manning books explain online evaluation?

AI Model Evaluation, Designing AI Systems and LLM Evaluation and Alignment, The Foundational Ideas.

What should I know before learning online evaluation?

Manning books that teach online evaluation usually introduce benchmark, accuracy, edge case and language model first.

What is the difference between online evaluation and offline evaluation?

Online evaluation is the assessment of a system using behavior, interactions, or outcomes observed during live operation. Offline evaluation is the assessment of a system using prepared or previously collected data outside live interactions or production traffic.

What kinds of online evaluation are there?

Kinds of online evaluation include A/B test.

What is online evaluation a kind of?

Online evaluation is a kind of evaluation. Evaluation is the systematic assessment of a system, model, process, or outcome against relevant criteria or objectives using evidence.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.