An evaluation suite is a set of assessments used to compare model responses against criteria across representative task contexts. Running the assessments after prompt or model changes can reveal regressions before they reach a product. section 7.10 section 7.12
Testing listing descriptions
A listing generator’s prompt is revised to improve accuracy and instruction-following. Running all the evaluations after each change helps catch improvements in one measure that come at the expense of another. section 7.10
Table 7.6 Methods of generative AI evaluations
| Type of evaluation |
When to use |
Advantages |
Disadvantages |
| Human preference or judgment |
Before an LLM judge is validated, or to confirm hypotheses about human preferences |
|
|
| Heuristic evaluation or verifier |
When “checking” the response or computing a metric can be done deterministically (such as with multiple-choice questions or measures like response length) |
|
|
| LLM as judge |
When the judgment of the response is for any nonverifiable or subjective quality |
|
|
| A/B test |
When interested in user preferences and behavior in a live production setting |
|
|
Compare human judgment, deterministic checks, and LLM judges by their use cases and tradeoffs when choosing evaluation methods. section 7.10
Evaluation scores do not necessarily show what users prefer; that is better measured through production A/B testing. LLM judges can also show biases or reward outputs that score well without meeting the intended goal, so their performance needs regular checking. section 7.10
