A benchmark measures and compares performance under defined conditions. It turns engineering claims into numbers that readers can inspect, question, and reproduce. section 14.0 section 14.6
Use the comparison’s relationship to the product to identify its type:
- Compare a company’s product with a competitor’s product. section 14.1.1
- Use a company’s product to compare another technology, such as cloud infrastructure or hardware. section 14.2
- Measure an independent technology or project without featuring the company’s product. section 14.1.3
Treat every result as guilty until proven innocent: state why conditions are fair, disclose hardware and software, show exact reproduction steps, and identify outside validators. Avoid apples-to-oranges comparisons, verify correctness, and warm caches unless cache warmup is the target. section 14.4.2 section 14.5.1
A benchmark readers can trust
A cloud database comparison reports processor, cores, memory, network, storage, prices, I/O results, test setup, six phases, charts, and tables before concluding that Graviton2 offers comparable performance at lower hourly cost. A C-versus-Rust comparison goes further by profiling cache misses and changing data structures to test the suspected cause. section 14.3.1 section 14.3.2
Lead with the important result, label chart directions such as “lower is better,” explain workload and configuration, and revise the analysis as critics expose missing tests or assumptions. section 14.3.3 section 14.6
