Online evaluation assesses live production traffic after deployment, connecting real user behavior to continuous improvement. It complements offline evaluation against curated datasets and human review: offline checks regressions early, online scoring exposes production gaps, and annotation supplies ground truth. section 8.3 section 8.3.3
A production scoring rule typically:
- Selects a configurable sample of completed production traces rather than evaluating every interaction. section 8.3.3
- Runs asynchronously after the trace completes, so evaluation does not delay the user response. section 8.3.3
- Combines LLM judges with automated checks, such as required elements or retrieval relevance. section 8.3.3
- Attaches scores to traces for dashboards and queries, then triggers alerts when scores remain below thresholds within a time window. section 8.3.3
- Feeds low-scoring and edge-case traces into evaluation datasets, where they can support later offline comparisons. section 8.6
Patient-intake monitoring
A scoring rule samples 10% of the patient-intake workflow, asks an LLM judge whether each response is helpful and actionable, checks for appointment, documents, and contact information, and evaluates retrieval relevance. It alerts when helpfulness falls below 0.6 or required elements fall below 0.7 over one hour. section 8.3.3
Figure 8.1 The improvement loop. Production traffic generates traces. Scores (automated, LLM-based, and human) attach quality signals. Low-scoring traces flow into evaluation datasets. Offline evals compare prompt versions. Winning versions enter A/B tests. Results feed back into production. This continuous cycle turns static AI deployments into systems that improve over time.

Trace how production interactions become scores, datasets, offline comparisons, experiments, and subsequent production changes. section 8.0
Listing 8.5 Online scoring rule for production traffic
platform.experiments.create_scoring_rule(
name="intake-helpfulness-monitor",
workflow_id="patient-intake",
sample_rate=0.10, #A
scorers=[
{"name": "helpfulness", "type": "llm_judge",
"judge_model": "claude-haiku-4-5",
"criterion": "Is the response helpful, specific, "
"and actionable for the patient?"},
{"name": "key_elements", "type": "automated",
"required_elements": ["appointment", "documents",
"contact"]}, #B
{"name": "retrieval_relevance",
"type": "automated"}, #C
],
alert_on={ #D
"helpfulness": {"below": 0.6, "window": "1h"},
"key_elements": {"below": 0.7, "window": "1h"},
},
)Inspect the scoring-rule configuration linking sampling, LLM-based evaluation, automated criteria, retrieval checks, and alert thresholds. section 8.3.3


