Recall is the proportion of actual positive cases that a classifier correctly identifies. It helps evaluate whether an LLM judge finds positive examples reliably, especially when overall accuracy could hide scarce “yes” labels. section 7.10
Recall appears in evaluation workflows that:
- Compare an LLM’s binary annotations with labels supplied by a human expert. section 7.10
- Assess whether an annotation prompt identifies the target quality with high recall and precision. section 7.10
- Support comparisons among future changes to a system prompt after the judge has been validated. section 7.10
Real estate listing evaluation
An LLM judge labels generated real estate descriptions for enthusiasm or positivity. Recall indicates how often it identifies descriptions that expert annotators marked positive. section 7.10
Table 7.5 Possible types of scores in LLM evaluations
| Score type |
Description |
Example prompt |
| Binary label |
Given an output and one or more criteria to evaluate, the LLM judge assesses whether or not the output meets those criteria. |
“Evaluate whether the output fulfills the user’s request. Answer yes or no.” |
| Classification |
Given an output and a set of categories, the LLM judge classifies the output into the relevant category. |
“Does the output 1) fulfill the user’s request completely (FULL), 2) fulfill the user’s request partially (PARTIAL), or 3) not fulfill the user’s request (NONE)? Answer FULL, PARTIAL, or NONE. |
| Numeric rating |
Given an output and a rating scale, the LLM judge assigns the output a rating. |
“On a scale of 1 to 5, how well does this output fulfill the user’s request?” |
| Ranking |
Given multiple outputs and a set of criteria, the LLM judge ranks the outputs in order of preference. |
“Which of these outputs best fulfills the user’s request?” |
Evaluation scores can use binary labels, classifications, numeric ratings, or rankings; recall applies when labels form a classifier-style evaluation. section 7.10
