Chapter 16: Evaluation

Introduction

Introduces evaluation as benchmark-based measurement of model quality and traces how evaluation priorities have changed alongside RLHF and post-training.

• Evaluation and benchmark purpose: Explain that evaluation uses techniques, commonly benchmarks, to assess model quality and the effects of training, and that a benchmark is a set of questions or environments designed to measure a particular property.
  Why it matters: This gives you a basis for interpreting what an evaluation result measures.
• Changing post-training evaluation priorities: Trace the shift from chat-focused comparisons, through evaluations spanning multiple skills, to harder reasoning and tool-use tasks. Explain that these regimes reflect prevailing post-training goals, and note both the scalability and length-bias risk of early LLM-as-a-judge evaluations.
  Why it matters: An evaluation result makes more sense when you know which capabilities its era prioritized and where its scoring could be biased.
• Expanding evaluation stakeholders: Describe how language-model evaluation has expanded beyond a single kind of benchmark provider to include private evaluations, community-driven evaluations, and third-party evaluation companies.
  Why it matters: Evaluation results now come from groups with different roles and incentives.

Key terms: evaluation, benchmarks, chat, LLM-as-a-judge, length bias, multiskill era, reasoning and tools, multistakeholder, private evaluations, community-driven evaluations

16.1 Prompting formatting

This section explains why prompt construction matters for evaluating language models, while noting that it is often easier to hurt a strong model’s performance than to improve it through prompting.

• Prompt Structure and Evaluation: Explain what a prompt provides to a language model and why changes to prompting can substantially alter measured evaluation performance.
  Why it matters: An evaluation result may reflect how the model was prompted, not just the model’s capabilities.
• Limits and Risks of Prompting: Distinguish prompting gains from improvements to data or training algorithms, and explain the section’s warning about strong models.
  Why it matters: Careful prompting helps preserve a strong model’s performance without mistaking prompt changes for improvements to the model itself.

Key terms: prompt, prompting techniques, evaluation, prompting, data, training algorithms

16.1.1 Few-shot prompting and log-likelihood scoring

This section explains how few-shot language-model evaluations prompt and score answers, and why formatting can distort the results.

• Few-shot evaluation setup: Describe how early language models were evaluated with several examples followed by an incomplete prompt, without instruction tuning or RLHF. Explain that the number of examples and the model's ability to follow their format affect the evaluation.
  Why it matters: The examples and expected format are part of what an evaluation is testing.
• Answer scoring methods: Distinguish scoring a generated answer by exact match from scoring candidate answers by their conditional log-likelihoods. Explain the difference between sampled answers and token probabilities, including the option to score an answer letter or the full answer text.
  Why it matters: The scoring method changes what is measured and whether repeated sampled attempts can differ.
• Evaluation format sensitivity: Explain why exact-match evaluation can understate a model's capability when its generated answer does not fit the expected format, and contrast the section's typical uses of exact match and log-likelihood scoring.
  Why it matters: A low score may reflect an answer-format bottleneck rather than the model's full ability on the task.

Key terms: few-shot prompting, in-context learning, exact match, majority voting, conditional log-likelihood scoring, log-probabilities, format-agnostic evaluation, pretraining evaluation, post-training

16.1.2 Chain-of-thought prompting

This subsection introduces chain-of-thought prompting as few-shot examples that show written-out reasoning for a model to follow.

• Reasoning in few-shot examples: Explain that chain-of-thought prompting here adds written-out reasoning to in-context examples for the model to follow, rather than merely instructing it to generate reasoning steps.
  Why it matters: It clarifies what changes in the prompt when reasoning is shown through examples.

Key terms: chain-of-thought examples, few-shot prompting, in-context examples, written-out reasoning

16.1.3 Zero-shot instruction following

This subsection traces the shift from zero-shot instruction following to generation-based evaluation as language models became more open-ended, while noting the continued use of multiple-choice tests.

• Zero-shot instruction generalization: Explain how fine-tuning on specific tasks could enable language models to answer questions they had not been trained on, and why the section identifies this as an early form of instruction fine-tuning.
  Why it matters: It explains why evaluations could begin asking models to handle questions beyond the tasks used to fine-tune them.
• Open-ended evaluation shift: Describe why sampling model responses became increasingly popular for evaluation as RLHF-era models supported more open-ended use, and how multiple-choice evaluation persisted.
  Why it matters: It helps explain why evaluation methods changed with model use without assuming older formats disappeared immediately.

Key terms: zero-shot evaluation, zero-shot learners, FLAN, instruction fine-tuning (IFT), evaluation with sampling from the model, generation-based [exact-match] evaluation, multiple-choice evaluations, temperature

16.1.4 Reasoning-era evaluation prompts

This section explains how reasoning models shifted evaluation from single-token answers to generated chain-of-thought responses checked for exact answers.

• Generative reasoning evaluation shift: Explain how evaluation changed for reasoning models: instead of sampling a single answer token, evaluators generate a response with chain-of-thought reasoning and check the final answer for an exact match.
  Why it matters: The way an answer is generated and checked changes what an evaluation measures.
• Reasoning prompt and output formats: Describe how reasoning models and their output formats affect evaluation prompts: they can reason without the canonical “think step by step” phrase, some setups use specially designed prompts, and thinking tokens may be separated from answer tokens.
  Why it matters: Prompt wording and response formatting can affect how an evaluator elicits and identifies a model’s answer.

Key terms: generation-based (exact-match) evaluation, CoT reasoning, single-token answer sampling, think step by step, specially designed prompts, thinking tokens, answer tokens

16.2 Why many external evaluation comparisons are unreliable

The section explains why published language-model evaluation scores can be difficult to reproduce or compare across labs and why benchmarks lose usefulness as they saturate.

• Limits of cross-lab comparisons: Explain why small differences between evaluation scores in model announcements should not be treated as decisive. Describe how undocumented, differing evaluation setups and unknown training-versus-test use limit comparisons.
  Why it matters: A score difference is less persuasive when the conditions that produced each score may differ.
• Inference token budget: Explain why comparisons should account for the number of tokens models use at inference time.
  Why it matters: Two scores may reflect different amounts of work done at inference time.
• Answer format sensitivity: Describe how post-training answer formats can affect evaluation performance, using the section’s math-data example.
  Why it matters: A benchmark’s requested answer format can favor one model’s training convention over another’s.
• Benchmark saturation: Explain what saturation means for an evaluation benchmark and why it weakens the benchmark as a measure of further progress.
  Why it matters: A nearly solved benchmark gives a less dependable picture of meaningful differences between models.

Key terms: evaluation setup, hillclimb, held-out evaluation sets, large error bars, inference-time scaling, total number of tokens for inference, evaluation formats, post-training, NuminaMath, MetaMath

16.3 How labs actually use evaluations internally to improve models

Explains how frontier labs use selected, often private evaluations and human feedback to compare training runs and guide model improvement.

• Independent test sets: Distinguish training on benchmark-related data from testing whether a model generalizes to the task, and explain why a lab must keep an evaluation independent if it uses it to track progress.
  Why it matters: A score means something different when the test has helped shape training.
• Focused internal progress signals: Explain how labs focus on selected evaluations, including nonpublic ones, and improve those evaluations to make comparisons between training runs more informative.
  Why it matters: The most useful internal evaluation is one that helps a lab tell whether a training change actually helped.
• Human post-training feedback: Describe how human evaluation contributes to post-training evaluation of generative models and reward models.
  Why it matters: Human comparisons supply signals that post-training evaluations cannot simply replace with benchmark scores.
• Generation-heavy evaluation costs: Describe why evaluating modern language models can require substantial computation, and identify one-off evaluation prompts as a reported way labs boost task performance.
  Why it matters: Evaluation choices affect both the resources a lab spends and the scores it obtains.

Key terms: true test set, generalizing, training sets, hillclimb, internal evaluations, statistical power, noise, human evaluation, Elo rankings, agreement

16.4 Contamination

This section explains how benchmark data can leak into language-model training, how overlaps are checked, and why suspected contamination is difficult to prove.

• Dataset contamination and decontamination: Explain what dataset contamination is, how benchmark material can enter training data, and how decontamination searches for overlap between training and evaluation datasets.
  Why it matters: A benchmark result is harder to interpret if its questions may have appeared in training.
• Near-benchmark training confounds: Describe how training on data close to, but not identical to, benchmark questions can complicate the interpretation of post-training results.
  Why it matters: An apparent training improvement may reflect prior exposure to similar problems.
• Perturbation benchmark evidence: Explain why slightly altered benchmark questions are used for models with undisclosed training data, and distinguish suggestive score variation from proof of contamination.
  Why it matters: Perturbed tests can raise useful questions about a model’s results without establishing that its training data contained the original benchmark.

Key terms: dataset contamination, data leakage, decontamination, n-gram overlap, base-model contamination, RLVR, confounding variable, perturbation benchmarks, contamination, high variance

16.5 Tooling

Lists open-source tools available for evaluating language models.

• Open-source evaluation tool options: Name examples of open-source evaluation tools listed in the section and explain that evaluators have multiple options to choose from.
  Why it matters: Knowing that several tools exist gives you a starting point for finding evaluation tooling.

Key terms: Inspect AI, LightEval, EleutherAI’s evaluation harness, HELM, Eval Gauntlet

16.6 Summary