liveBook · Manning

Online evaluation

As explained in Designing AI Systems

Online evaluation assesses an AI system or change using live production interactions and their outcomes.

Online evaluation assesses live production traffic after deployment, connecting real user behavior to continuous improvement. It complements offline evaluation against curated datasets and human review: offline checks regressions early, online scoring exposes production gaps, and annotation supplies ground truth. section 8.3 section 8.3.3

A production scoring rule typically:

  • Selects a configurable sample of completed production traces rather than evaluating every interaction. section 8.3.3
  • Runs asynchronously after the trace completes, so evaluation does not delay the user response. section 8.3.3
  • Combines LLM judges with automated checks, such as required elements or retrieval relevance. section 8.3.3
  • Attaches scores to traces for dashboards and queries, then triggers alerts when scores remain below thresholds within a time window. section 8.3.3
  • Feeds low-scoring and edge-case traces into evaluation datasets, where they can support later offline comparisons. section 8.6

Patient-intake monitoring

A scoring rule samples 10% of the patient-intake workflow, asks an LLM judge whether each response is helpful and actionable, checks for appointment, documents, and contact information, and evaluates retrieval relevance. It alerts when helpfulness falls below 0.6 or required elements fall below 0.7 over one hour. section 8.3.3

Figure 8.1 The improvement loop. Production traffic generates traces. Scores (automated, LLM-based, and human) attach quality signals. Low-scoring traces flow into evaluation datasets. Offline evals compare prompt versions. Winning versions enter A/B tests. Results feed back into production. This continuous cycle turns static AI deployments into systems that improve over time.

Trace how production interactions become scores, datasets, offline comparisons, experiments, and subsequent production changes. section 8.0

Listing 8.5 Online scoring rule for production traffic
platform.experiments.create_scoring_rule(
    name="intake-helpfulness-monitor",
    workflow_id="patient-intake",
    sample_rate=0.10,    #A
    scorers=[
        {"name": "helpfulness", "type": "llm_judge",
         "judge_model": "claude-haiku-4-5",
         "criterion": "Is the response helpful, specific, "
                     "and actionable for the patient?"},
        {"name": "key_elements", "type": "automated",
         "required_elements": ["appointment", "documents",
                              "contact"]},    #B
        {"name": "retrieval_relevance",
         "type": "automated"},    #C
    ],
    alert_on={    #D
        "helpfulness": {"below": 0.6, "window": "1h"},
        "key_elements": {"below": 0.7, "window": "1h"},
    },
)

Inspect the scoring-rule configuration linking sampling, LLM-based evaluation, automated criteria, retrieval checks, and alert thresholds. section 8.3.3

How Designing AI Systems covers it

How deep it goes
Defines asynchronous sampling, scorer configuration, trace attachment, dashboards, queries, and alerts, but not scorer implementation internals.
Examples
Uses a sampled patient-intake workflow with helpfulness, required-elements, retrieval-relevance, and threshold-based alert scorers.
What it assumes
Assumes completed production traces and introduces how automated, LLM-based, and human signals reinforce evaluation datasets and improvement.
What you can do after
Operate/configure it through scoring rules that sample live traffic, attach quality signals, expose trends, and alert on degradation.
How it connects
Connects production traces with offline evaluation, human evaluation, evaluation datasets, observability, A/B testing, and alerts.

Also explained in

Compare how 3 books explain online evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.