liveBook · Manning

Benchmark

A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.

How 30 Manning books explain it

Generative AI in Action

Benchmarking is the systematic measurement and comparison of models or systems against defined tasks, datasets, or criteria.

How it is used
Guides model selection, fine-tuning comparisons, and production readiness for enterprise GenAI applications.
How deep it goes
Explains benchmark components and evaluation steps, then compares named benchmarks and their limitations.
Examples
Illustrates translation baseline comparisons, common-sense multiple-choice questions, and multimodal questions using charts and diagrams.
What you can do after
Evaluate and understand model performance against a baseline using task-relevant criteria and complementary reviews.

Read how Generative AI in Action explains benchmark →

Effective Data Analysis

A benchmark is a reference value or standard used to evaluate and compare an observed measurement or proportion.

How it is used
Uses benchmarks to anchor organizational metric targets in previously achieved performance.
How deep it goes
Defines benchmarking and outlines reference sources and considerations for choosing feasible targets.
Examples
Compares customer churn with an industry benchmark and cites diagnostic accuracy and sales win rates as potential benchmarks.
What you can do after
Evaluate and understand metric performance against comparable references when setting achievable targets.

Read how Effective Data Analysis explains benchmark →

Rearchitecting LLMs

A benchmark is a standardized task or test suite used to measure and compare model performance.

How it is used
Tracks general capability preservation alongside task-specific quality and inference efficiency during model specialization.
How deep it goes
Details test-set isolation, metric selection, harness validation, baseline recording, and repeated comparison after model changes.
Examples
Compares Gemma 3 270M before pruning, after pruning, and after distillation on ARC Easy, WinoGrande, HellaSwag, and LAMBADA.
What you can do after
Evaluate and understand model changes using task-specific and general benchmarks alongside hardware efficiency measurements.

Read how Rearchitecting LLMs explains benchmark →

Demand Forecasting Best Practices

A benchmark is a reference forecasting model or comparison value used to judge the performance of another forecasting model or process.

How it is used
Judges model and demand-planning process accuracy against a simple, fair reference forecast.
How deep it goes
Explains benchmark selection, lag matching, overfitting avoidance, and continuous process comparison without model internals.
Examples
Compares forecast MAE and bias against naïve and six-period moving-average forecasts, then contrasts a process with its benchmark.
What you can do after
Evaluate/understand whether a forecast or process adds accuracy beyond a suitable benchmark.

Read how Demand Forecasting Best Practices explains benchmark →

Writing for Developers: Blogs that get read

The systematic measurement and comparison of a system's performance under defined conditions.

How it is used
Uses measured comparisons to substantiate engineering claims, educate readers, or promote a product.
How deep it goes
Requires reproducible conditions, environment details, correctness checks, causal analysis, and clearly reported results.
Examples
Uses cloud database comparisons, C-versus-Rust rewrites, Redpanda-versus-Kafka tests, and io_uring comparisons.
What you can do after
Evaluate/understand performance claims and design reports that readers can reproduce and interpret.

Read how Writing for Developers: Blogs that get read explains benchmark →

AI Agents in Action, Second Edition

A benchmark test measures how well a system solves a defined problem or achieves a known goal using a repeatable evaluation set.

How it is used
Uses benchmarks to define agent goals and guide iterative testing and development.
How deep it goes
Works through benchmark cases, repeated runs, evaluation choices, and responses to conflicting goals.
Examples
Tests a retrieval agent using fictional knowledge-base questions with expected and incorrect answers.
What you can do after
Apply benchmarks to an agent by defining cases, rerunning tests, and iteratively changing prompts or tools.

Read how AI Agents in Action, Second Edition explains benchmark →

100 Go Mistakes and How to Avoid Them

A benchmark is a controlled measurement of software execution under specified workloads or inputs.

How it is used
Uses benchmarks to compare Go implementation choices and check optimization claims.
How deep it goes
Explains benchmark iteration, timer control, measurement variability, and compiler optimization traps with corrected examples.
Examples
Compares slice initialization strategies, atomic stores, and a population-count function.
What you can do after
Apply benchmarking via Go's testing package and benchstat to compare implementations while guarding against common measurement errors.

Read how 100 Go Mistakes and How to Avoid Them explains benchmark →

Domain-Specific Small Language Models

A standardized evaluation set or task suite used to compare model performance under common conditions.

How it is used
Guides model-version selection and checks real serving performance before client access.
How deep it goes
Works through warmup and repeated timing, then specifies serving workloads and report metrics; treats generated-result quality separately.
Examples
Compares GPT-2 execution variants on CPU and benchmarks Phi-3 Mini requests served by vLLM with ShareGPT prompts.
What you can do after
Apply it via a timing helper or vLLM script to compare candidates and inspect deployed-request performance.

Read how Domain-Specific Small Language Models explains benchmark →

Go by Example

A benchmark is a repeatable performance measurement that runs a piece of code, usually many times, to assess its execution characteristics.

How it is used
Measures code performance repeatedly to track regressions, identify bottlenecks, and guide justified optimization.
How deep it goes
Explains benchmark construction, metrics, sub-benchmarks, comparisons, and compiler-optimization pitfalls; defers profiling details.
Examples
Uses URL.String across varying URL lengths, plus a pure Pow function eliminated by compiler optimization.
What you can do after
Apply it via Go’s testing package to measure operations, allocations, and comparative implementation changes.

Read how Go by Example explains benchmark →

Latency

A benchmark is a repeatable performance measurement procedure applied to a defined workload or operation.

How it is used
Quantifies implementation and concurrency trade-offs against defined workloads and baselines.
How deep it goes
Walks through benchmark setup, repeated timing, comparison, and CPU profiling, with sampling warnings and outliers.
Examples
Measures Fibonacci implementations, sequential versus concurrent delayed tasks, and shared-counter locks at different thread counts.
What you can do after
Apply it via Criterion to measure and compare small Rust workloads, and use CPU profiling to investigate runtime.

Read how Latency explains benchmark →

Vibe Engineering

A benchmark is a standardized evaluation procedure that measures a system against defined inputs and expected results.

How it is used
Measures LLM-generated SQL, coding capability, and performance while checking whether benchmark results answer the intended engineering question.
How deep it goes
Details a SQL evaluation pipeline and reporting, while deferring low-level implementation methods to the repository.
Examples
Uses HumanEval, EvalPlus, SWE-bench, SQL generation, and account-search performance comparisons as benchmark examples.
What you can do after
Evaluate and understand benchmark scores by relating their inputs and tests to the risks left unmeasured.

Read how Vibe Engineering explains benchmark →

LLMs in Production

A benchmark dataset is a curated dataset used to evaluate and compare model performance on defined tasks.

How it is used
Structures repeatable model comparisons by pairing defined tasks with metrics and reference answers.
How deep it goes
Defines benchmark datasets, discusses custom and responsible variants, but stops short of a complete curation procedure.
Examples
Uses SQuAD, custom edge-case prompts, WinoBias, CALM, WinoQueer, and multiple-choice evaluation datasets.
What you can do after
Evaluate models with the Evaluate library, compare task results, and identify capability, bias, and coverage gaps.

Read how LLMs in Production explains benchmark →

Sutskever's List

A benchmark is a defined evaluation dataset or task used to compare the capabilities or performance of models.

How it is used
Measures model performance on defined tasks as a practical proxy for capability, not as a complete account of intelligence.
How deep it goes
Explains saturation, contamination, and limits on interpreting scores, but does not lay out a benchmark-construction procedure.
Examples
Uses ImageNet, SQuAD, GLUE, and public tests including ARC-AGI, GPQA–Diamond, SWE-bench, and FrontierMath.
What you can do after
Evaluate model capability against agreed task criteria without inferring human-like understanding or intrinsic intelligence from scores.

Read how Sutskever's List explains benchmark →

The Art of AI Product Development

A benchmark is a standardized dataset, task, or evaluation procedure used to compare model capabilities or performance.

How it is used
Supports language-model selection, release thresholds, optimization, and comparison before deployment.
How deep it goes
Defines public and custom benchmarks, names task metrics, and stops short of treating leaderboard results as sufficient evidence.
Examples
Uses MMLU, HELM, ARC, SQuAD, and QuAC to illustrate broad, reasoning, and question-answering evaluation.
What you can do after
Evaluate language models with public benchmarks, then extend comparisons using custom metrics and real-world application tests.

Read how The Art of AI Product Development explains benchmark →

Graph Neural Networks in Action

Benchmarking is the systematic measurement of a system or model against representative data to establish performance references.

How it is used
Guides project planning by establishing resource and training-performance baselines before large-scale experimentation.
How deep it goes
Provides a practical measurement workflow and troubleshooting example, but does not specify statistical benchmarking methodology.
Examples
Uses CPU-versus-GPU timing, a 10 GB sparse graph, subgraph sampling, and Open Graph Benchmark datasets.
What you can do after
Evaluate hardware and GNN configurations using representative data and measurements such as memory, epoch time, FLOP/s, and convergence time.

Read how Graph Neural Networks in Action explains benchmark →

Reinforcement Learning from Human Feedback

A benchmark is a standardized set of questions, tasks, or environments designed to measure a specific property of a model.

How it is used
Measures specific model properties through standardized questions or environments, providing signals for post-training evaluation.
How deep it goes
Defines benchmarks and explains score confounders, but does not specify a general benchmark-construction procedure.
Examples
Uses MT-Bench, AlpacaEval, MMLU, GPQA, SWE-Bench, MATH, and AIME as examples.
What you can do after
Evaluate model results as rough indicators while accounting for prompt sensitivity, cross-lab variation, and contamination.

Read how Reinforcement Learning from Human Feedback explains benchmark →

Learn Haskell by Example

Benchmarking is the systematic measurement of program performance under specified conditions.

How it is used
Uses runtime measurements to verify performance gains while tuning parallel image-processing parameters.
How deep it goes
Explains comparative timing and spark statistics, but leaves benchmarking as programmer-guided parameter tweaking.
Examples
Compares faulty and chunked vector strategies using spark counts and image-processing execution times.
What you can do after
Diagnose parallel performance and tune chunk parameters using execution times and runtime spark statistics.

Read how Learn Haskell by Example explains benchmark →

Build an AI Agent (From Scratch)

A benchmark is a standardized evaluation used to compare system performance on a specified set of tasks.

How it is used
Uses benchmarks to set measurable agent-development targets, establish baselines, and compare systems.
How deep it goes
Defines benchmark selection criteria and shows dataset loading, but stops short of demonstrating a complete evaluation run.
Examples
Illustrates benchmarks with GAIA, SWE-bench, HotpotQA, and a research-and-calculation question about marathon pace and lunar distance.
What you can do after
Evaluate/understand baseline agent capability and cross-system results while recognizing that benchmark scores may not predict product performance.

Read how Build an AI Agent (From Scratch) explains benchmark →

Learn Go with Pocket-Sized Projects

Benchmarking is the measurement and comparison of program performance across implementations or versions.

How it is used
Measures algorithm, implementation, version, and concurrency performance during development and regression checking.
How deep it goes
Explains comparison and regression purposes, then qualifies measurements by workload type and machine architecture.
Examples
Uses algorithm comparisons, commit checks, buffered-channel sizing, remote files, and CPU-intensive video encoding.
What you can do after
Evaluate performance with Go's built-in bench command and diagnose whether concurrency changes improve or degrade a workload.

Read how Learn Go with Pocket-Sized Projects explains benchmark →

.NET in Action, Second Edition

A benchmark is a standardized or repeatable performance measurement of a component, operation, or system.

How it is used
Measures code performance to guide .NET optimization, using comparisons to answer questions with data rather than relying on absolute timings.
How deep it goes
Builds a BenchmarkDotNet comparison, explains returning values to resist compiler removal, and notes environment-dependent timings.
Examples
Compares Newtonsoft.Json and System.Text.Json deserialization of a jokes.json file and reports their measured means.
What you can do after
Apply BenchmarkDotNet to compare implementations and use relative measurements to inform performance decisions.

Read how .NET in Action, Second Edition explains benchmark →

Effective Conversational AI

Benchmarking is the systematic measurement and comparison of a system against defined performance criteria or alternatives.

How it is used
Measures whether RAG changes improve retrieval accuracy, response relevance, and system performance.
How deep it goes
Outlines dataset and query selection, configuration comparison, and performance metrics without detailing benchmark execution.
Examples
Uses PharmaBot queries and compares indexing and embedding options across accuracy, relevance, latency, throughput, and resource metrics.
What you can do after
Evaluate RAG configurations against user-focused and operational metrics.

Read how Effective Conversational AI explains benchmark →

Go in Action, Second Edition

Benchmark testing measures how code performs under repeated execution with controlled inputs and conditions.

How it is used
Uses repeated execution to compare algorithm performance and target areas for improvement.
How deep it goes
Explains testing.B, b.N, result validation, command execution, and output fields, but does not develop profiling internals.
Examples
Demonstrates naive byte-slice and strings.Builder random-string generators with paired benchmark functions and command output.
What you can do after
Apply it via Go's built-in testing library to establish baselines and compare function implementations using measured operation times.

Read how Go in Action, Second Edition explains benchmark →

The Art of Code

A microbenchmark is a controlled performance measurement of a small code operation or component.

How it is used
Measures small code snippets in isolation to investigate runtime efficiency and environmental impact.
How deep it goes
Defines isolated measurement and names frameworks, then defers deeper CPU, allocation, and memory-layout analysis to profilers and JOL.
Examples
Names JMH, BenchmarkDotNet, pytest-benchmark, language-specific profilers, and the Java Object Layout toolkit.
What you can do after
Apply it via JMH, BenchmarkDotNet, or pytest-benchmark to measure small operations, then use profilers for deeper diagnosis.

Read how The Art of Code explains benchmark →

Retrieval Augmented Generation, The Foundational Ideas

A benchmark is a standardized evaluation task, dataset, or reference result used to compare system performance.

How it is used
Supports reliable comparison of retrieval and RAG systems across quality, ranking, dataset coverage, and deployment efficiency.
How deep it goes
Explains metric interpretation, baselines, splits, query distributions, and efficiency trade-offs, but does not detail benchmark implementation.
Examples
Uses Natural Questions, TriviaQA, BEIR, MTEB, HotpotQA, MS MARCO, and SQuAD as evaluation references.
What you can do after
Evaluate retrieval systems by comparing metrics, baselines, test splits, query types, and operational costs.

Read how Retrieval Augmented Generation, The Foundational Ideas explains benchmark →

AI Model Evaluation

A fixed reference dataset or evaluation standard used to compare the performance of different models or approaches.

How it is used
Enables offline comparison of model versions before online serving.
How deep it goes
Defines holdout construction and production caveats, but stops short of specifying evaluation metrics.
Examples
Uses recent interaction data split into training, validation, and recent-weeks holdout datasets.
What you can do after
Evaluate model versions offline and identify why static results may not transfer to production.

Read how AI Model Evaluation explains benchmark →

Idiomatic Rust

A benchmark is a repeatable performance measurement of a program operation under specified conditions.

How it is used
Compares collection performance across append, search, and removal workloads to inform data-structure choices.
How deep it goes
Shows benchmark functions and measured results for three operations across Vec, HashSet, and LinkedList.
Examples
Times appending a million elements, finding random values, and removing elements from collections.
What you can do after
Evaluate collection choices by running Rust benchmarks with cargo bench and comparing workload timings.

Read how Idiomatic Rust explains benchmark →

Starting Data Analytics with Generative AI and Python

Benchmarking is the comparison of a system's performance with a reference implementation or established standard metrics.

How it is used
Uses benchmarking to compare analytics-system performance with a reference implementation or industry-standard metrics.
How deep it goes
Defines benchmarking within performance measurement and prioritizes execution time over throughput and resource utilization.
Examples
Uses computationally intensive bootstrap confidence-interval calculations for average sales price and review score by state.
What you can do after
Evaluate analytics-system efficiency by comparing performance against a reference or established metrics.

Read how Starting Data Analytics with Generative AI and Python explains benchmark →

Fast Python: High performance techniques for large datasets

How it is used
Uses benchmarks to choose NumPy distribution components and compare data-writing approaches.
How deep it goes
Compares timings and disk use with concrete tests, while noting buffering and data-pattern effects.
Examples
Compares NumPy and Blosc writes for zero-filled, tiled, and random arrays.
What you can do after
Evaluate implementations by timing code-specific tests and comparing elapsed time and storage use.

Read how Fast Python: High performance techniques for large datasets explains benchmark →

CUDA for LLMs

A benchmark is a reference workload or measurement basis used to compare performance or correctness.

How it is used
Establishes a trusted PyTorch reference for checking whether lower-level neural-network implementations train correctly and run faster.
How deep it goes
Measures loss across epochs and times five training stages, while allowing for nonidentical random initialization.
Examples
Uses MNIST digit classification and timed PyTorch, NumPy, and C training runs.
What you can do after
Evaluate correctness through comparable loss curves and locate performance costs by timed training stage.

Read how CUDA for LLMs explains benchmark →

Machines that Think

A standardized task, dataset, or procedure used to compare system performance.

How it is used
Compares language-model performance across defined tasks, practical work, and human judgments of usefulness.
How deep it goes
Distinguishes three evaluation approaches and their limits, without detailing benchmark construction or scoring procedures.
Examples
Uses MMLU, GSM8K, HumanEval, SWE-bench, AgentBench, GAIA, MT-Bench, and Chatbot Arena as examples.
What you can do after
Evaluate model performance across different tasks while recognizing that one benchmark or score cannot capture every weakness.

Read how Machines that Think explains benchmark →

Also covered in

What to know first

Related concepts

Questions about benchmark

What is benchmark?

A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.

Which Manning books explain benchmark?

Generative AI in Action, Effective Data Analysis, Rearchitecting LLMs, Demand Forecasting Best Practices, Writing for Developers: Blogs that get read, AI Agents in Action, Second Edition, 100 Go Mistakes and How to Avoid Them, Domain-Specific Small Language Models and 22 more.

What should I know before learning benchmark?

Manning books that teach benchmark usually introduce language model, performance, accuracy and prompt first.

What is the difference between benchmark and baseline?

A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions. A baseline is a reference model, result, measurement, or level used to compare performance or assess change.

What kinds of benchmark are there?

Kinds of benchmark include MMLU, BBQ, HEIM and MMMU.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.