Automated scoring evaluates recorded outputs with deterministic heuristic code rather than generating a judgment. It is cheap, runs in milliseconds, and suits structural checks such as required elements, valid JSON, and length limits. section 7.6.1
Treat the scorer as a separate evaluation component that reads a trace and emits a typed score. A score identifies the trace, optional span or generation, measured property, value, source, metadata, and timestamp. section 7.6.1 section 7.2.2
Production deployments use automated scoring for broad, low-cost coverage:
- Run heuristic rules on every trace when lookup or computation is inexpensive. section 7.6.1 section 10.10
- Represent results as numeric, categorical, or Boolean values and mark the source as automated. section 7.6.1
- Combine heuristics with sampled model judges and human labels; use human reviews to calibrate whether automated rules catch real problems. section 7.6.1 section 8.1
- Run scoring asynchronously after delivery so quality signals accumulate without blocking the response path. section 7.2.2
Key-element completeness
A KeyElementScorer lowercases the response, counts required elements found in the output, and returns found divided by the number of required elements. A patient-intake trace might check for “electronic records,” “patient portal,” and “photo ID.” section 7.6.1 section 7.2.2
Figure 7.8 The three sources of quality scores and their calibration relationship. Automated scorers run on all traffic at near-zero cost. LLM judges sample 5-20% of traffic at moderate cost. Human reviewers label 1-5% at highest accuracy. Human scores calibrate both LLM judges (do they agree on helpfulness?) and automated scorers (are the heuristics catching real problems?). This layered approach gives broad coverage with targeted depth.

Use the layered scoring diagram to compare automated coverage with sampled model judgments and human calibration. section 7.6.1
