16 Evaluation
This chapter covers
- Evaluating language models during and after RLHF
- The evolution from chat-focused to reasoning-focused evaluations
- Why prompting format dramatically affects benchmark performance
- The unreliability of evaluation comparisons across labs
- How dataset contamination confounds results
Evaluation is the set of techniques used to understand the quality and influence of the training processes detailed in this book. It is normally expressed through benchmarks (examples of popular benchmarks include Massive Multitask Language Understanding [MMLU], Graduate-Level Google-Proof Q&A [GPQA], SWE-Bench [SWE is short for software engineering], and MATH [named after the subject]), which are discrete sets of questions or environments designed to measure a specific property of a model.] Evaluation is an ever-evolving approach; this chapter presents vignettes of popular evaluation regimes throughout the early history of RLHF so you can understand the details, the failure modes, and the common themes that will carry forward into the future of language modeling.
The key to understanding language model evaluation, particularly with post-training, is that the current popular evaluation regimes represent a reflection of the popular training best practices and goals. Although challenging evaluations drive progress in language models to new areas, the majority of evaluation is designed around building useful signals for new models.