LLM-as-a-judge uses a large language model to assess, score, rank, or compare model outputs against defined criteria. It is a scoring mechanism, not a metric category: the metric might be relevance, groundedness, or instruction following. The judgment is an evaluation signal, not absolute truth. section 9.0 section 1.3.4
A judge is useful when quality depends on meaning, context, or subjective comparison and applying a defined judgment manually is too slow or costly. Use exact matching, rules, or simpler metrics when they answer the question. Judge scores complement deterministic metrics and live evaluation; they do not establish real-world product impact on their own. section 9.11 section 1.4
Design the evaluation around a decision, not just a prompt:
- Define the goal and quality criteria. Decide what the result will inform and whether you need scores, a ranking, or a pairwise winner. section 9.4 section 9.5.3
- Select representative examples and give the judge the context needed to assess them, such as the user query, source document, retrieved evidence, and candidate output. section 9.4.2 section 9.5.2
- Write an explicit rubric and a machine-readable output schema. Score separate dimensions before combining them into an overall judgment. section 9.5.2
- Calibrate with known examples, then validate against human ratings, gold labels, or known cases. Inspect disagreements and test whether the judge is reliable enough for the intended decision. section 9.11 section 9.5.2
Judging movie recommendations
To evaluate a recommender’s top five movies, give the judge the user’s viewing history and recommended titles, then define relevance and diversity criteria. A vague request to judge how good the recommendations are can produce inconsistent scales and unstructured explanations. An explicit rubric and output format make results easier to compare and inspect. section 9.6.1 section 9.5.2
Choose the format to match the decision. Scores support aggregate trends but need calibration; rankings compare several candidates but can be sensitive to order and style. Pairwise comparisons are useful for comparing variants, although comparing many candidates can require many judge calls. section 9.5.3
Test for position, verbosity, and style bias, as well as insufficient context, prompt injection, and confident explanations of questionable scores. Consistency alone is not proof of correctness. High-impact decisions still need human or domain-specific review. section 9.7 section 9.3 section 9.11
Treat the judge as an operational system. Choose a model for instruction following, structured output, cost, latency, and the task’s context needs. Version prompts and log the evaluator configuration, rubric, dataset snapshot, sampling strategy, and calibration results. Monitor malformed outputs and drift so that comparisons remain interpretable over time. section 9.5.4 section 9.10.2 section 9.11




















