A successful, fast request can still produce an unhelpful or inaccurate response. A quality score records how well an operation or output meets an evaluation criterion, making quality observable alongside latency, errors, and cost. section 7.1.1 section 7.2.1
Scores attach to traces, spans, or generations. They can be numeric, categorical, or boolean, and can come from an automated heuristic, a model-based evaluator, or a human reviewer. Record the source with each score so automated judgments can be calibrated against human judgments over time. section 7.2.1
The observability interface separates score recording and querying into RecordScore and QueryScores. Production scoring can run asynchronously after a trace completes, keeping evaluation off the response path. Resulting scores can feed dashboards and alerts and support comparisons between quality and cost. section 7.3.1 section 8.3.3 section 7.1.3
Listing 8.5 Online scoring rule for production traffic
platform.experiments.create_scoring_rule(
name="intake-helpfulness-monitor",
workflow_id="patient-intake",
sample_rate=0.10, #A
scorers=[
{"name": "helpfulness", "type": "llm_judge",
"judge_model": "claude-haiku-4-5",
"criterion": "Is the response helpful, specific, "
"and actionable for the patient?"},
{"name": "key_elements", "type": "automated",
"required_elements": ["appointment", "documents",
"contact"]}, #B
{"name": "retrieval_relevance",
"type": "automated"}, #C
],
alert_on={ #D
"helpfulness": {"below": 0.6, "window": "1h"},
"key_elements": {"below": 0.7, "window": "1h"},
},
)This production rule samples 10% of patient intake traces, combines a model judge with automated checks, and sets alert thresholds for helpfulness and required elements. section 8.3.3
Sampling is an operational tradeoff: an automated retrieval-relevance scorer can run on every trace using an existing span value, while a model judge can sample 10% to limit evaluation cost. Scores accumulated over time can reveal low-quality question categories or degradation as conversations grow. section 10.10









