liveBook · Manning

Automated scoring

As explained in Designing AI Systems

An automated scorer is a non-generative evaluation procedure that computes response-quality scores from recorded metrics or rules.

Automated scoring evaluates recorded outputs with deterministic heuristic code rather than generating a judgment. It is cheap, runs in milliseconds, and suits structural checks such as required elements, valid JSON, and length limits. section 7.6.1

Treat the scorer as a separate evaluation component that reads a trace and emits a typed score. A score identifies the trace, optional span or generation, measured property, value, source, metadata, and timestamp. section 7.6.1 section 7.2.2

Production deployments use automated scoring for broad, low-cost coverage:

  • Run heuristic rules on every trace when lookup or computation is inexpensive. section 7.6.1 section 10.10
  • Represent results as numeric, categorical, or Boolean values and mark the source as automated. section 7.6.1
  • Combine heuristics with sampled model judges and human labels; use human reviews to calibrate whether automated rules catch real problems. section 7.6.1 section 8.1
  • Run scoring asynchronously after delivery so quality signals accumulate without blocking the response path. section 7.2.2

Key-element completeness

A KeyElementScorer lowercases the response, counts required elements found in the output, and returns found divided by the number of required elements. A patient-intake trace might check for “electronic records,” “patient portal,” and “photo ID.” section 7.6.1 section 7.2.2

Figure 7.8 The three sources of quality scores and their calibration relationship. Automated scorers run on all traffic at near-zero cost. LLM judges sample 5-20% of traffic at moderate cost. Human reviewers label 1-5% at highest accuracy. Human scores calibrate both LLM judges (do they agree on helpfulness?) and automated scorers (are the heuristics catching real problems?). This layered approach gives broad coverage with targeted depth.

Use the layered scoring diagram to compare automated coverage with sampled model judgments and human calibration. section 7.6.1

How Designing AI Systems covers it

How deep it goes
Defines score schemas, trace linkage, asynchronous execution, and completeness logic, but defers subjective quality to judges or humans.
Examples
Uses required-element completeness, valid-JSON checks, retrieval relevance, patient-intake facts, and sampled response-quality monitoring.
What it assumes
Assumes trace data and recorded outputs, while establishing how to attach typed quality scores to requests, spans, or generations.
What you can do after
Apply it via scoring rules and scorer implementations that produce queryable quality signals for every eligible trace.
How it connects
Connects traces, observability, LLM-as-judge evaluation, human review, experimentation, calibration, and quality monitoring.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.