14 The “Benchmarks and Test Results” pattern

This chapter covers

  • Blog posts that share various flavors of benchmarks and tests
  • Their purpose and audience
  • How various authors approached this type of post
  • Key elements of successful “Benchmarks and Test Results” posts
  • Dos and don’ts for your own “Benchmarks and Test Results” post

Skepticism about benchmarks is so trendy right now that “All benchmarks are wrong” could become an entire pattern chapter in this book. This herd skepticism against benchmarks is actually a positive trend. Why not pay closer attention to whether benchmarks and test results shared in engineering blogs truly make sense? On the flip side, writers beware. If you’re writing one of these blog posts, it’s more important than ever to do your homework and ensure that the published benchmarks are truthful, reproducible, and not too synthetic.

“Benchmarks and Test Results” articles tend to fall into at least three distinct categories:

  • Tests that compare the company’s product against its competition
  • Tests that compare something (such as cloud infrastructure or hardware) using the company’s product
  • Measuring something independent of the company’s products (e.g., an open source project or a new graphics processing unit)

14.1 Purpose

14.1.1 Benchmarketing

14.1.2 Subtle benchmarketing

14.1.3 Community service

14.2 Audience

14.3 Examples of “Benchmarks and Test Results” blog posts

14.3.1 AWS Graviton2: Arm Brings Better Price-Performance than Intel

14.3.2 The Relative Performance of C and Rust

14.3.3 Redpanda vs. Kafka: A Performance Comparison

14.3.4 The Effect of Switching to TCMalloc on RocksDB Memory Use

14.3.5 How Much Does Rust’s Bounds Checking Actually Cost?

14.4 Characteristics

14.4.1 Numeric and visual

14.4.2 Guilty until proven innocent