liveBook · Manning

Quality

As explained in Designing AI Systems

A score is a numerical or categorical value representing how well an observed operation or output performed against an evaluation criterion.

A successful, fast request can still produce an unhelpful or inaccurate response. A quality score records how well an operation or output meets an evaluation criterion, making quality observable alongside latency, errors, and cost. section 7.1.1 section 7.2.1

Scores attach to traces, spans, or generations. They can be numeric, categorical, or boolean, and can come from an automated heuristic, a model-based evaluator, or a human reviewer. Record the source with each score so automated judgments can be calibrated against human judgments over time. section 7.2.1

The observability interface separates score recording and querying into RecordScore and QueryScores. Production scoring can run asynchronously after a trace completes, keeping evaluation off the response path. Resulting scores can feed dashboards and alerts and support comparisons between quality and cost. section 7.3.1 section 8.3.3 section 7.1.3

Listing 8.5 Online scoring rule for production traffic
platform.experiments.create_scoring_rule(
    name="intake-helpfulness-monitor",
    workflow_id="patient-intake",
    sample_rate=0.10,    #A
    scorers=[
        {"name": "helpfulness", "type": "llm_judge",
         "judge_model": "claude-haiku-4-5",
         "criterion": "Is the response helpful, specific, "
                     "and actionable for the patient?"},
        {"name": "key_elements", "type": "automated",
         "required_elements": ["appointment", "documents",
                              "contact"]},    #B
        {"name": "retrieval_relevance",
         "type": "automated"},    #C
    ],
    alert_on={    #D
        "helpfulness": {"below": 0.6, "window": "1h"},
        "key_elements": {"below": 0.7, "window": "1h"},
    },
)

This production rule samples 10% of patient intake traces, combines a model judge with automated checks, and sets alert thresholds for helpfulness and required elements. section 8.3.3

Sampling is an operational tradeoff: an automated retrieval-relevance scorer can run on every trace using an existing span value, while a model judge can sample 10% to limit evaluation cost. Scores accumulated over time can reveal low-quality question categories or degradation as conversations grow. section 10.10

How Designing AI Systems covers it

How deep it goes
Defines score values, sources, and attachment points, then configures asynchronous scoring rules, sampling, and alerts.
Examples
Configures a patient intake rule with a helpfulness judge, required-element checks, retrieval relevance, and alert thresholds.
What you can do after
Operate/configure production scoring rules with sampling, evaluators, and alert thresholds.
How it connects
Connects quality scoring to distributed tracing, model-based evaluation, retrieval relevance, cost attribution, and experimentation.

Also explained in

Compare how 10 books explain quality →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.