A/B testing measures user preferences by comparing two versions under real production conditions. A system offers different models or prompts to groups of users, then compares outcomes such as engagement, satisfaction, or other chosen metrics. section 7.10
This approach answers questions that response scoring alone cannot, especially whether users prefer one version. It should complement human judgment, deterministic checks, or an LLM judge rather than replace them. Automated judges can reward superficial cues instead of intended quality, so evaluators require regular checking. section 7.10 section 7.10
Table 7.6 Methods of generative AI evaluations
| Type of evaluation |
When to use |
Advantages |
Disadvantages |
| Human preference or judgment |
Before an LLM judge is validated, or to confirm hypotheses about human preferences |
|
|
| Heuristic evaluation or verifier |
When “checking” the response or computing a metric can be done deterministically (such as with multiple-choice questions or measures like response length) |
|
|
| LLM as judge |
When the judgment of the response is for any nonverifiable or subjective quality |
|
|
| A/B test |
When interested in user preferences and behavior in a live production setting |
|
|
Compare A/B testing with human preference judgments, heuristic evaluation, and LLM-based judging to see how each method balances scalability, cost, and reliability. section 7.10
