liveBook · Manning

Evaluation

As explained in Machines that Think

Evaluation is the systematic assessment of a model's outputs or capabilities against specified criteria or reference results.

Evaluation assesses a model’s outputs or capabilities against defined criteria. The choice of criteria matters: a convincing answer is not necessarily reliable, and observable performance alone does not establish understanding. Evaluation therefore informs how much trust, verification, and oversight a particular use requires. section 1.1.4

Different benchmarks test different aspects of usefulness. Some measure correctness on well-defined questions; others test whether outputs work in context or whether people prefer them. These approaches reflect a shift from treating an answer as a final product to evaluating work that may be refined through interaction. section 8.2.1

Examples include:

  • MMLU tests knowledge across academic subjects, while GSM8K tests grade-school math reasoning. section 8.2.1
  • HumanEval tests functional code generation, SWE-bench tests resolution of repository issues, and AgentBench and GAIA assess multi-step tasks. section 8.2.1
  • MT-Bench compares response quality, while Chatbot Arena ranks models through user preference comparisons. section 8.2.1

What did the puzzle measure?

In Tower of Hanoi tests, requiring a complete move sequence made long-output generation part of the task: truncation or formatting drift could cause failure even if a model knew the rule. Later analysis still found difficulty with incremental, interactive prompting, suggesting weaknesses in state tracking and execution. In River Crossing, some harder instances were reportedly unsolvable, so an evaluation would need to credit recognition of impossibility. Interface restrictions also matter: a text-only test without tools measures something different from task-solving with external support. section 9.5.1

Evaluation criteria can also shape behavior. When accuracy on attempted answers is rewarded but refusal receives little or no reward, evaluation and fine-tuning can favor answering over acknowledging uncertainty. Critical evaluation asks not only how often a model succeeds, but what the setup rewards and what a failure actually reveals. section 6.1.2 section 9.5.1

The distinction between an output and what produced it also matters in education. When a student can obtain a correct answer with AI assistance, the answer alone may no longer demonstrate understanding; attempts to resolve this through AI-writing detectors can themselves produce inconsistent results and false positives. section 8.3.1

How Machines that Think covers it

How deep it goes
Compares benchmark goals and examines how puzzle design, output requirements, and interfaces complicate interpretation of failures.
Examples
Illustrates evaluation with MMLU, GSM8K, SWE-bench, Chatbot Arena, Tower of Hanoi, and River Crossing.
What you can do after
Evaluate model results by distinguishing observed performance from the effects of scoring, task design, and available tools.
How it connects
Connects evaluation with benchmarking, human preference, reasoning, hallucination, and fine-tuning.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.