An LLM judge uses a language model to assess another model’s outputs against specified criteria. It can automate judgments that human annotators would otherwise make, helping teams evaluate prompt changes more systematically. section 7.10
A judge’s labels should be compared with expert human labels before trusting it on new responses. Agreement can be assessed with precision and recall; accuracy alone may mislead when examples are unevenly distributed across labels. Automated judging can scale more readily than human review, but it does not make subjective judgments infallible. section 7.10
Real estate listing descriptions
A judge can classify whether a listing description is appropriately enthusiastic and positive. Comparing its yes-or-no labels with an expert realtor’s labels helps assess whether it can evaluate new descriptions similarly. section 7.10
Table 7.5 Possible types of scores in LLM evaluations
| Score type |
Description |
Example prompt |
| Binary label |
Given an output and one or more criteria to evaluate, the LLM judge assesses whether or not the output meets those criteria. |
“Evaluate whether the output fulfills the user’s request. Answer yes or no.” |
| Classification |
Given an output and a set of categories, the LLM judge classifies the output into the relevant category. |
“Does the output 1) fulfill the user’s request completely (FULL), 2) fulfill the user’s request partially (PARTIAL), or 3) not fulfill the user’s request (NONE)? Answer FULL, PARTIAL, or NONE. |
| Numeric rating |
Given an output and a rating scale, the LLM judge assigns the output a rating. |
“On a scale of 1 to 5, how well does this output fulfill the user’s request?” |
| Ranking |
Given multiple outputs and a set of criteria, the LLM judge ranks the outputs in order of preference. |
“Which of these outputs best fulfills the user’s request?” |
Evaluation scores can take forms including binary labels, categories, numeric ratings, or rankings. section 7.10




















