Benchmark
A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.
Generative AI in Action
Benchmarking is the systematic measurement and comparison of models or systems against defined tasks, datasets, or criteria.
- How it is used
- Guides model selection, fine-tuning comparisons, and production readiness for enterprise GenAI applications.
- How deep it goes
- Explains benchmark components and evaluation steps, then compares named benchmarks and their limitations.
- Examples
- Illustrates translation baseline comparisons, common-sense multiple-choice questions, and multimodal questions using charts and diagrams.
- What you can do after
- Evaluate and understand model performance against a baseline using task-relevant criteria and complementary reviews.
Effective Data Analysis
A benchmark is a reference value or standard used to evaluate and compare an observed measurement or proportion.
- How it is used
- Uses benchmarks to anchor organizational metric targets in previously achieved performance.
- How deep it goes
- Defines benchmarking and outlines reference sources and considerations for choosing feasible targets.
- Examples
- Compares customer churn with an industry benchmark and cites diagnostic accuracy and sales win rates as potential benchmarks.
- What you can do after
- Evaluate and understand metric performance against comparable references when setting achievable targets.
Rearchitecting LLMs
A benchmark is a standardized task or test suite used to measure and compare model performance.
- How it is used
- Tracks general capability preservation alongside task-specific quality and inference efficiency during model specialization.
- How deep it goes
- Details test-set isolation, metric selection, harness validation, baseline recording, and repeated comparison after model changes.
- Examples
- Compares Gemma 3 270M before pruning, after pruning, and after distillation on ARC Easy, WinoGrande, HellaSwag, and LAMBADA.
- What you can do after
- Evaluate and understand model changes using task-specific and general benchmarks alongside hardware efficiency measurements.
Demand Forecasting Best Practices
A benchmark is a reference forecasting model or comparison value used to judge the performance of another forecasting model or process.
- How it is used
- Judges model and demand-planning process accuracy against a simple, fair reference forecast.
- How deep it goes
- Explains benchmark selection, lag matching, overfitting avoidance, and continuous process comparison without model internals.
- Examples
- Compares forecast MAE and bias against naïve and six-period moving-average forecasts, then contrasts a process with its benchmark.
- What you can do after
- Evaluate/understand whether a forecast or process adds accuracy beyond a suitable benchmark.
Read how Demand Forecasting Best Practices explains benchmark →
Writing for Developers: Blogs that get read
The systematic measurement and comparison of a system's performance under defined conditions.
- How it is used
- Uses measured comparisons to substantiate engineering claims, educate readers, or promote a product.
- How deep it goes
- Requires reproducible conditions, environment details, correctness checks, causal analysis, and clearly reported results.
- Examples
- Uses cloud database comparisons, C-versus-Rust rewrites, Redpanda-versus-Kafka tests, and io_uring comparisons.
- What you can do after
- Evaluate/understand performance claims and design reports that readers can reproduce and interpret.
Read how Writing for Developers: Blogs that get read explains benchmark →
AI Agents in Action, Second Edition
A benchmark test measures how well a system solves a defined problem or achieves a known goal using a repeatable evaluation set.
- How it is used
- Uses benchmarks to define agent goals and guide iterative testing and development.
- How deep it goes
- Works through benchmark cases, repeated runs, evaluation choices, and responses to conflicting goals.
- Examples
- Tests a retrieval agent using fictional knowledge-base questions with expected and incorrect answers.
- What you can do after
- Apply benchmarks to an agent by defining cases, rerunning tests, and iteratively changing prompts or tools.
Read how AI Agents in Action, Second Edition explains benchmark →
100 Go Mistakes and How to Avoid Them
A benchmark is a controlled measurement of software execution under specified workloads or inputs.
- How it is used
- Uses benchmarks to compare Go implementation choices and check optimization claims.
- How deep it goes
- Explains benchmark iteration, timer control, measurement variability, and compiler optimization traps with corrected examples.
- Examples
- Compares slice initialization strategies, atomic stores, and a population-count function.
- What you can do after
- Apply benchmarking via Go's testing package and benchstat to compare implementations while guarding against common measurement errors.
Read how 100 Go Mistakes and How to Avoid Them explains benchmark →
Domain-Specific Small Language Models
A standardized evaluation set or task suite used to compare model performance under common conditions.
- How it is used
- Guides model-version selection and checks real serving performance before client access.
- How deep it goes
- Works through warmup and repeated timing, then specifies serving workloads and report metrics; treats generated-result quality separately.
- Examples
- Compares GPT-2 execution variants on CPU and benchmarks Phi-3 Mini requests served by vLLM with ShareGPT prompts.
- What you can do after
- Apply it via a timing helper or vLLM script to compare candidates and inspect deployed-request performance.
Read how Domain-Specific Small Language Models explains benchmark →
Go by Example
A benchmark is a repeatable performance measurement that runs a piece of code, usually many times, to assess its execution characteristics.
- How it is used
- Measures code performance repeatedly to track regressions, identify bottlenecks, and guide justified optimization.
- How deep it goes
- Explains benchmark construction, metrics, sub-benchmarks, comparisons, and compiler-optimization pitfalls; defers profiling details.
- Examples
- Uses URL.String across varying URL lengths, plus a pure Pow function eliminated by compiler optimization.
- What you can do after
- Apply it via Go’s testing package to measure operations, allocations, and comparative implementation changes.
Latency
A benchmark is a repeatable performance measurement procedure applied to a defined workload or operation.
- How it is used
- Quantifies implementation and concurrency trade-offs against defined workloads and baselines.
- How deep it goes
- Walks through benchmark setup, repeated timing, comparison, and CPU profiling, with sampling warnings and outliers.
- Examples
- Measures Fibonacci implementations, sequential versus concurrent delayed tasks, and shared-counter locks at different thread counts.
- What you can do after
- Apply it via Criterion to measure and compare small Rust workloads, and use CPU profiling to investigate runtime.
Vibe Engineering
A benchmark is a standardized evaluation procedure that measures a system against defined inputs and expected results.
- How it is used
- Measures LLM-generated SQL, coding capability, and performance while checking whether benchmark results answer the intended engineering question.
- How deep it goes
- Details a SQL evaluation pipeline and reporting, while deferring low-level implementation methods to the repository.
- Examples
- Uses HumanEval, EvalPlus, SWE-bench, SQL generation, and account-search performance comparisons as benchmark examples.
- What you can do after
- Evaluate and understand benchmark scores by relating their inputs and tests to the risks left unmeasured.
LLMs in Production
A benchmark dataset is a curated dataset used to evaluate and compare model performance on defined tasks.
- How it is used
- Structures repeatable model comparisons by pairing defined tasks with metrics and reference answers.
- How deep it goes
- Defines benchmark datasets, discusses custom and responsible variants, but stops short of a complete curation procedure.
- Examples
- Uses SQuAD, custom edge-case prompts, WinoBias, CALM, WinoQueer, and multiple-choice evaluation datasets.
- What you can do after
- Evaluate models with the Evaluate library, compare task results, and identify capability, bias, and coverage gaps.
Sutskever's List
A benchmark is a defined evaluation dataset or task used to compare the capabilities or performance of models.
- How it is used
- Measures model performance on defined tasks as a practical proxy for capability, not as a complete account of intelligence.
- How deep it goes
- Explains saturation, contamination, and limits on interpreting scores, but does not lay out a benchmark-construction procedure.
- Examples
- Uses ImageNet, SQuAD, GLUE, and public tests including ARC-AGI, GPQA–Diamond, SWE-bench, and FrontierMath.
- What you can do after
- Evaluate model capability against agreed task criteria without inferring human-like understanding or intrinsic intelligence from scores.
The Art of AI Product Development
A benchmark is a standardized dataset, task, or evaluation procedure used to compare model capabilities or performance.
- How it is used
- Supports language-model selection, release thresholds, optimization, and comparison before deployment.
- How deep it goes
- Defines public and custom benchmarks, names task metrics, and stops short of treating leaderboard results as sufficient evidence.
- Examples
- Uses MMLU, HELM, ARC, SQuAD, and QuAC to illustrate broad, reasoning, and question-answering evaluation.
- What you can do after
- Evaluate language models with public benchmarks, then extend comparisons using custom metrics and real-world application tests.
Read how The Art of AI Product Development explains benchmark →
Graph Neural Networks in Action
Benchmarking is the systematic measurement of a system or model against representative data to establish performance references.
- How it is used
- Guides project planning by establishing resource and training-performance baselines before large-scale experimentation.
- How deep it goes
- Provides a practical measurement workflow and troubleshooting example, but does not specify statistical benchmarking methodology.
- Examples
- Uses CPU-versus-GPU timing, a 10 GB sparse graph, subgraph sampling, and Open Graph Benchmark datasets.
- What you can do after
- Evaluate hardware and GNN configurations using representative data and measurements such as memory, epoch time, FLOP/s, and convergence time.
Read how Graph Neural Networks in Action explains benchmark →
Reinforcement Learning from Human Feedback
A benchmark is a standardized set of questions, tasks, or environments designed to measure a specific property of a model.
- How it is used
- Measures specific model properties through standardized questions or environments, providing signals for post-training evaluation.
- How deep it goes
- Defines benchmarks and explains score confounders, but does not specify a general benchmark-construction procedure.
- Examples
- Uses MT-Bench, AlpacaEval, MMLU, GPQA, SWE-Bench, MATH, and AIME as examples.
- What you can do after
- Evaluate model results as rough indicators while accounting for prompt sensitivity, cross-lab variation, and contamination.
Read how Reinforcement Learning from Human Feedback explains benchmark →
Learn Haskell by Example
Benchmarking is the systematic measurement of program performance under specified conditions.
- How it is used
- Uses runtime measurements to verify performance gains while tuning parallel image-processing parameters.
- How deep it goes
- Explains comparative timing and spark statistics, but leaves benchmarking as programmer-guided parameter tweaking.
- Examples
- Compares faulty and chunked vector strategies using spark counts and image-processing execution times.
- What you can do after
- Diagnose parallel performance and tune chunk parameters using execution times and runtime spark statistics.
Build an AI Agent (From Scratch)
A benchmark is a standardized evaluation used to compare system performance on a specified set of tasks.
- How it is used
- Uses benchmarks to set measurable agent-development targets, establish baselines, and compare systems.
- How deep it goes
- Defines benchmark selection criteria and shows dataset loading, but stops short of demonstrating a complete evaluation run.
- Examples
- Illustrates benchmarks with GAIA, SWE-bench, HotpotQA, and a research-and-calculation question about marathon pace and lunar distance.
- What you can do after
- Evaluate/understand baseline agent capability and cross-system results while recognizing that benchmark scores may not predict product performance.
Read how Build an AI Agent (From Scratch) explains benchmark →
Learn Go with Pocket-Sized Projects
Benchmarking is the measurement and comparison of program performance across implementations or versions.
- How it is used
- Measures algorithm, implementation, version, and concurrency performance during development and regression checking.
- How deep it goes
- Explains comparison and regression purposes, then qualifies measurements by workload type and machine architecture.
- Examples
- Uses algorithm comparisons, commit checks, buffered-channel sizing, remote files, and CPU-intensive video encoding.
- What you can do after
- Evaluate performance with Go's built-in bench command and diagnose whether concurrency changes improve or degrade a workload.
Read how Learn Go with Pocket-Sized Projects explains benchmark →
.NET in Action, Second Edition
A benchmark is a standardized or repeatable performance measurement of a component, operation, or system.
- How it is used
- Measures code performance to guide .NET optimization, using comparisons to answer questions with data rather than relying on absolute timings.
- How deep it goes
- Builds a BenchmarkDotNet comparison, explains returning values to resist compiler removal, and notes environment-dependent timings.
- Examples
- Compares Newtonsoft.Json and System.Text.Json deserialization of a jokes.json file and reports their measured means.
- What you can do after
- Apply BenchmarkDotNet to compare implementations and use relative measurements to inform performance decisions.
Read how .NET in Action, Second Edition explains benchmark →
Effective Conversational AI
Benchmarking is the systematic measurement and comparison of a system against defined performance criteria or alternatives.
- How it is used
- Measures whether RAG changes improve retrieval accuracy, response relevance, and system performance.
- How deep it goes
- Outlines dataset and query selection, configuration comparison, and performance metrics without detailing benchmark execution.
- Examples
- Uses PharmaBot queries and compares indexing and embedding options across accuracy, relevance, latency, throughput, and resource metrics.
- What you can do after
- Evaluate RAG configurations against user-focused and operational metrics.
Go in Action, Second Edition
Benchmark testing measures how code performs under repeated execution with controlled inputs and conditions.
- How it is used
- Uses repeated execution to compare algorithm performance and target areas for improvement.
- How deep it goes
- Explains testing.B, b.N, result validation, command execution, and output fields, but does not develop profiling internals.
- Examples
- Demonstrates naive byte-slice and strings.Builder random-string generators with paired benchmark functions and command output.
- What you can do after
- Apply it via Go's built-in testing library to establish baselines and compare function implementations using measured operation times.
The Art of Code
A microbenchmark is a controlled performance measurement of a small code operation or component.
- How it is used
- Measures small code snippets in isolation to investigate runtime efficiency and environmental impact.
- How deep it goes
- Defines isolated measurement and names frameworks, then defers deeper CPU, allocation, and memory-layout analysis to profilers and JOL.
- Examples
- Names JMH, BenchmarkDotNet, pytest-benchmark, language-specific profilers, and the Java Object Layout toolkit.
- What you can do after
- Apply it via JMH, BenchmarkDotNet, or pytest-benchmark to measure small operations, then use profilers for deeper diagnosis.
Retrieval Augmented Generation, The Foundational Ideas
A benchmark is a standardized evaluation task, dataset, or reference result used to compare system performance.
- How it is used
- Supports reliable comparison of retrieval and RAG systems across quality, ranking, dataset coverage, and deployment efficiency.
- How deep it goes
- Explains metric interpretation, baselines, splits, query distributions, and efficiency trade-offs, but does not detail benchmark implementation.
- Examples
- Uses Natural Questions, TriviaQA, BEIR, MTEB, HotpotQA, MS MARCO, and SQuAD as evaluation references.
- What you can do after
- Evaluate retrieval systems by comparing metrics, baselines, test splits, query types, and operational costs.
Read how Retrieval Augmented Generation, The Foundational Ideas explains benchmark →
AI Model Evaluation
A fixed reference dataset or evaluation standard used to compare the performance of different models or approaches.
- How it is used
- Enables offline comparison of model versions before online serving.
- How deep it goes
- Defines holdout construction and production caveats, but stops short of specifying evaluation metrics.
- Examples
- Uses recent interaction data split into training, validation, and recent-weeks holdout datasets.
- What you can do after
- Evaluate model versions offline and identify why static results may not transfer to production.
Idiomatic Rust
A benchmark is a repeatable performance measurement of a program operation under specified conditions.
- How it is used
- Compares collection performance across append, search, and removal workloads to inform data-structure choices.
- How deep it goes
- Shows benchmark functions and measured results for three operations across Vec, HashSet, and LinkedList.
- Examples
- Times appending a million elements, finding random values, and removing elements from collections.
- What you can do after
- Evaluate collection choices by running Rust benchmarks with cargo bench and comparing workload timings.
Starting Data Analytics with Generative AI and Python
Benchmarking is the comparison of a system's performance with a reference implementation or established standard metrics.
- How it is used
- Uses benchmarking to compare analytics-system performance with a reference implementation or industry-standard metrics.
- How deep it goes
- Defines benchmarking within performance measurement and prioritizes execution time over throughput and resource utilization.
- Examples
- Uses computationally intensive bootstrap confidence-interval calculations for average sales price and review score by state.
- What you can do after
- Evaluate analytics-system efficiency by comparing performance against a reference or established metrics.
Read how Starting Data Analytics with Generative AI and Python explains benchmark →
Fast Python: High performance techniques for large datasets
- How it is used
- Uses benchmarks to choose NumPy distribution components and compare data-writing approaches.
- How deep it goes
- Compares timings and disk use with concrete tests, while noting buffering and data-pattern effects.
- Examples
- Compares NumPy and Blosc writes for zero-filled, tiled, and random arrays.
- What you can do after
- Evaluate implementations by timing code-specific tests and comparing elapsed time and storage use.
Read how Fast Python: High performance techniques for large datasets explains benchmark →
CUDA for LLMs
A benchmark is a reference workload or measurement basis used to compare performance or correctness.
- How it is used
- Establishes a trusted PyTorch reference for checking whether lower-level neural-network implementations train correctly and run faster.
- How deep it goes
- Measures loss across epochs and times five training stages, while allowing for nonidentical random initialization.
- Examples
- Uses MNIST digit classification and timed PyTorch, NumPy, and C training runs.
- What you can do after
- Evaluate correctness through comparable loss curves and locate performance costs by timed training stage.
Machines that Think
A standardized task, dataset, or procedure used to compare system performance.
- How it is used
- Compares language-model performance across defined tasks, practical work, and human judgments of usefulness.
- How deep it goes
- Distinguishes three evaluation approaches and their limits, without detailing benchmark construction or scoring procedures.
- Examples
- Uses MMLU, GSM8K, HumanEval, SWE-bench, AgentBench, GAIA, MT-Bench, and Chatbot Arena as examples.
- What you can do after
- Evaluate model performance across different tasks while recognizing that one benchmark or score cannot capture every weakness.
Also covered in
- Essential GraphRAG
- Agent Design PatternsIn early access (MEAP)
- Pandas Workout: 200 exercises to make you a stronger data analyst
- Robotics for ProgrammersIn early access (MEAP)
- The AI Pocket Book
- Data-Driven Cybersecurity
- Go in Practice, Second Edition
- Python Concurrency with asyncio
- AI-Powered Search
- Refactoring to Rust
- Building Specialized AI SystemsIn early access (MEAP)
- Deep Learning with Python, Third Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- performancePerformance is a measure of how effectively a person, process, model, or system achieves its objectives, including qualities such as output quality, speed, and resource efficiency.
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- promptA prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
Related concepts
- MMLUincludes · MMLU is a benchmark of questions spanning diverse academic and professional subjects, used to evaluate language models’ broad knowledge and reasoning.
- baselinecontrasts with · A baseline is a reference model, result, measurement, or level used to compare performance or assess change.
- BBQincludes · BBQ (Bias Benchmark for QA) is a question-answering benchmark dataset for evaluating social biases in language models, including responses to ambiguous and disambiguated contexts.
- go testuses · go test is the Go command that builds and runs tests for specified packages and can also execute benchmarks and report coverage.
Questions about benchmark
What is benchmark?
A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions.
Which Manning books explain benchmark?
Generative AI in Action, Effective Data Analysis, Rearchitecting LLMs, Demand Forecasting Best Practices, Writing for Developers: Blogs that get read, AI Agents in Action, Second Edition, 100 Go Mistakes and How to Avoid Them, Domain-Specific Small Language Models and 22 more.
What should I know before learning benchmark?
Manning books that teach benchmark usually introduce language model, performance, accuracy and prompt first.
What is the difference between benchmark and baseline?
A benchmark is a defined, repeatable task, workload, test, dataset, or reference standard used to measure or compare systems or methods under specified conditions. A baseline is a reference model, result, measurement, or level used to compare performance or assess change.
What kinds of benchmark are there?
Kinds of benchmark include MMLU, BBQ, HEIM and MMMU.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





























