liveBook · Manning

LLM-as-a-judge

As explained in Introduction to Generative AI, Second Edition

An LLM judge is a language model configured to assess or classify the outputs of another model according to specified criteria.

An LLM judge uses a language model to assess another model’s outputs against specified criteria. It can automate judgments that human annotators would otherwise make, helping teams evaluate prompt changes more systematically. section 7.10

A judge’s labels should be compared with expert human labels before trusting it on new responses. Agreement can be assessed with precision and recall; accuracy alone may mislead when examples are unevenly distributed across labels. Automated judging can scale more readily than human review, but it does not make subjective judgments infallible. section 7.10

Real estate listing descriptions

A judge can classify whether a listing description is appropriately enthusiastic and positive. Comparing its yes-or-no labels with an expert realtor’s labels helps assess whether it can evaluate new descriptions similarly. section 7.10

Table 7.5 Possible types of scores in LLM evaluations

Score type

Description

Example prompt

Binary label

Given an output and one or more criteria to evaluate, the LLM judge assesses whether or not the output meets those criteria.

“Evaluate whether the output fulfills the user’s request. Answer yes or no.”

Classification

Given an output and a set of categories, the LLM judge classifies the output into the relevant category.

“Does the output 1) fulfill the user’s request completely (FULL), 2) fulfill the user’s request partially (PARTIAL), or 3) not fulfill the user’s request (NONE)? Answer FULL, PARTIAL, or NONE.

Numeric rating

Given an output and a rating scale, the LLM judge assigns the output a rating.

“On a scale of 1 to 5, how well does this output fulfill the user’s request?”

Ranking

Given multiple outputs and a set of criteria, the LLM judge ranks the outputs in order of preference.

“Which of these outputs best fulfills the user’s request?”

Evaluation scores can take forms including binary labels, categories, numeric ratings, or rankings. section 7.10

How Introduction to Generative AI, Second Edition covers it

How deep it goes
Works through expert-label validation and precision and recall, while noting that accuracy can mislead with imbalanced labels.
Examples
Uses real estate listing descriptions and judges whether their tone is appropriately enthusiastic and positive.
What you can do after
Evaluate whether a judge’s labels resemble expert labels by checking agreement, precision, and recall.
How it connects
Connects LLM judging to human annotation, classifier evaluation, precision and recall, and knowledge distillation.

Also explained in

Compare how 21 books explain LLM-as-a-judge →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Introduction to Generative AI, Second Edition. The section links point to the parts of the book it draws on.