An LLM judge scores one model’s output using another model. Use it for qualities that are difficult to capture with simple checks, such as helpfulness, tone, and conversational flow. Each evaluation costs a model call, may vary between runs, and can carry bias—including a tendency to favor the judge’s own style. section 7.6.1
Treat the judge as one scorer in an observability pipeline: a shared scorer interface accepts a trace and returns a score, which can be associated with a trace, span, or generation. The score records its name, value, and source, distinguishing model judgments from automated checks and human reviews. section 7.6.1
Operationally, combine scoring methods rather than relying on the judge alone:
- Run deterministic, low-cost heuristic checks broadly for structural requirements and factual completeness. section 7.6.1
- Sample production traffic for LLM judging when evaluating subjective qualities; account for its cost, variability, and potential bias. section 7.6.1
- Use human reviews on a smaller sample to provide ground truth and calibrate both model-based and automated scores. section 7.6.1
Figure 7.8 The three sources of quality scores and their calibration relationship. Automated scorers run on all traffic at near-zero cost. LLM judges sample 5-20% of traffic at moderate cost. Human reviewers label 1-5% at highest accuracy. Human scores calibrate both LLM judges (do they agree on helpfulness?) and automated scorers (are the heuristics catching real problems?). This layered approach gives broad coverage with targeted depth.

The diagram compares the coverage and cost of automated scoring, LLM judging, and human review, and shows their calibration relationship. section 7.6.1




















