Evaluation assesses a model’s outputs or capabilities against defined criteria. The choice of criteria matters: a convincing answer is not necessarily reliable, and observable performance alone does not establish understanding. Evaluation therefore informs how much trust, verification, and oversight a particular use requires. section 1.1.4
Different benchmarks test different aspects of usefulness. Some measure correctness on well-defined questions; others test whether outputs work in context or whether people prefer them. These approaches reflect a shift from treating an answer as a final product to evaluating work that may be refined through interaction. section 8.2.1
Examples include:
- MMLU tests knowledge across academic subjects, while GSM8K tests grade-school math reasoning. section 8.2.1
- HumanEval tests functional code generation, SWE-bench tests resolution of repository issues, and AgentBench and GAIA assess multi-step tasks. section 8.2.1
- MT-Bench compares response quality, while Chatbot Arena ranks models through user preference comparisons. section 8.2.1
What did the puzzle measure?
In Tower of Hanoi tests, requiring a complete move sequence made long-output generation part of the task: truncation or formatting drift could cause failure even if a model knew the rule. Later analysis still found difficulty with incremental, interactive prompting, suggesting weaknesses in state tracking and execution. In River Crossing, some harder instances were reportedly unsolvable, so an evaluation would need to credit recognition of impossibility. Interface restrictions also matter: a text-only test without tools measures something different from task-solving with external support. section 9.5.1
Evaluation criteria can also shape behavior. When accuracy on attempted answers is rewarded but refusal receives little or no reward, evaluation and fine-tuning can favor answering over acknowledging uncertainty. Critical evaluation asks not only how often a model succeeds, but what the setup rewards and what a failure actually reveals. section 6.1.2 section 9.5.1
The distinction between an output and what produced it also matters in education. When a student can obtain a correct answer with AI assistance, the answer alone may no longer demonstrate understanding; attempts to resolve this through AI-writing detectors can themselves produce inconsistent results and false positives. section 8.3.1




























