liveBook · Manning

LLM-as-a-judge

As explained in AI Model Evaluation

An evaluation method in which a large language model assesses another model's outputs or behaviors and assigns judgments or scores.

LLM-as-a-judge uses a large language model to assess, score, rank, or compare model outputs against defined criteria. It is a scoring mechanism, not a metric category: the metric might be relevance, groundedness, or instruction following. The judgment is an evaluation signal, not absolute truth. section 9.0 section 1.3.4

A judge is useful when quality depends on meaning, context, or subjective comparison and applying a defined judgment manually is too slow or costly. Use exact matching, rules, or simpler metrics when they answer the question. Judge scores complement deterministic metrics and live evaluation; they do not establish real-world product impact on their own. section 9.11 section 1.4

Design the evaluation around a decision, not just a prompt:

  • Define the goal and quality criteria. Decide what the result will inform and whether you need scores, a ranking, or a pairwise winner. section 9.4 section 9.5.3
  • Select representative examples and give the judge the context needed to assess them, such as the user query, source document, retrieved evidence, and candidate output. section 9.4.2 section 9.5.2
  • Write an explicit rubric and a machine-readable output schema. Score separate dimensions before combining them into an overall judgment. section 9.5.2
  • Calibrate with known examples, then validate against human ratings, gold labels, or known cases. Inspect disagreements and test whether the judge is reliable enough for the intended decision. section 9.11 section 9.5.2

Judging movie recommendations

To evaluate a recommender’s top five movies, give the judge the user’s viewing history and recommended titles, then define relevance and diversity criteria. A vague request to judge how good the recommendations are can produce inconsistent scales and unstructured explanations. An explicit rubric and output format make results easier to compare and inspect. section 9.6.1 section 9.5.2

Choose the format to match the decision. Scores support aggregate trends but need calibration; rankings compare several candidates but can be sensitive to order and style. Pairwise comparisons are useful for comparing variants, although comparing many candidates can require many judge calls. section 9.5.3

Test for position, verbosity, and style bias, as well as insufficient context, prompt injection, and confident explanations of questionable scores. Consistency alone is not proof of correctness. High-impact decisions still need human or domain-specific review. section 9.7 section 9.3 section 9.11

Treat the judge as an operational system. Choose a model for instruction following, structured output, cost, latency, and the task’s context needs. Version prompts and log the evaluator configuration, rubric, dataset snapshot, sampling strategy, and calibration results. Monitor malformed outputs and drift so that comparisons remain interpretable over time. section 9.5.4 section 9.10.2 section 9.11

How AI Model Evaluation covers it

How deep it goes
Develops a procedural evaluation workflow from goal and rubric design through calibration, validation, model selection, and operational monitoring.
Examples
Uses a movie recommender’s top-five outputs and viewing history to illustrate relevance, diversity, and an underspecified judging prompt.
What you can do after
Evaluate and understand judge-assisted assessments by designing rubrics, choosing comparison formats, and validating judgments against known examples or human ratings.
How it connects
Connects offline evaluation, deterministic metrics, human ratings, retrieval grounding, and A/B testing.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.