liveBook · Manning

Benchmark

As explained in Machines that Think

A standardized task, dataset, or procedure used to compare system performance.

A benchmark is a task, dataset, or procedure used to compare system performance. For language models, what counts as performance has broadened: evaluation can test isolated correctness, practical task performance, or human judgments of usefulness. section 8.2.1 section 2.6.6

Common benchmark approaches emphasize different kinds of evidence:

  • Correctness on defined tasks: MMLU tests academic knowledge, while GSM8K tests step-by-step reasoning on grade-school math problems. section 8.2.1
  • Performance in practical contexts: HumanEval tests functional code generation, SWE-bench tests issue resolution in software repositories, and AgentBench or GAIA assess multistep tasks involving tools and extended reasoning. section 8.2.1
  • Human preferences: MT-Bench compares response quality, while Chatbot Arena ranks models through user preference comparisons. section 8.2.1

No single score captures language quality. A model may be fluent but factually wrong, or perform well on one benchmark while failing on another dimension. Improvements on one measure can therefore conceal weaknesses elsewhere; evaluation is more informative when it reflects the varied demands of actual tasks. section 2.6.6 section 8.2.1

In generative AI workflows, benchmark results are one guide to capability, not a guarantee that an initial output is ready to use. Outputs may need domain-specific refinement and human judgment. section 8.2.1

How Machines that Think covers it

How deep it goes
Distinguishes three evaluation approaches and their limits, without detailing benchmark construction or scoring procedures.
Examples
Uses MMLU, GSM8K, HumanEval, SWE-bench, AgentBench, GAIA, MT-Bench, and Chatbot Arena as examples.
What you can do after
Evaluate model performance across different tasks while recognizing that one benchmark or score cannot capture every weakness.
How it connects
Connects large language models with task performance, human preference, and output refinement.

Also explained in

Compare how 30 books explain benchmark →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.