Custom benchmarking code measures application performance without a dedicated benchmarking harness. It can produce useful comparisons, but plausible timings are not necessarily reliable: inadequate warm-up and JVM optimizations can distort results, while unrealistic input sizes can make the benchmark answer the wrong question. section 7.1.2
Account lookup comparison
An AI-generated AccountFinderPerformanceTest compared single-threaded and concurrent lookup strategies using 100,000 accounts without establishing that this matched production. Reducing the count to 10,000 better reflected the expected workload. The custom results favored the single-threaded version, but measurement uncertainty remained; one concurrent implementation also called indexOf inside a grouping operation, introducing quadratic work. section 7.1.2
public class AccountFinderPerformanceTest {
private AccountFinder accountFinder;
private List<Account> accounts;
private static final int ACCOUNT_COUNT = 100_000; #A
private static final int SEARCH_COUNT = 1000;
@BeforeEach
void setUp() {
accounts = generateAccounts(ACCOUNT_COUNT);
accountFinder = new AccountFinder(accounts);
}
@Test
void testPerformanceComparison() {
Random random = new Random();
List<Integer> searchIds = generateRandomIds(random, SEARCH_COUNT);
warmUp(); #B
long singleThreadTime = measureSingleThreadedPerformance(searchIds);
#C
System.out.println("Single-threaded time: " + singleThreadTime + " ms");
long parallelTime = measureParallelPerformance(searchIds); #D
System.out.println("Parallel stream time: " + parallelTime + " ms");
long concurrentTime = measureConcurrentPerformance(searchIds); #E
System.out.println("Concurrent time: " + concurrentTime + " ms");
long optimizedTime = measureOptimizedPerformance(searchIds); #F
System.out.println("Optimized time: " + optimizedTime + " ms");
verifyResultsConsistency(searchIds); #G
assertTrue(parallelTime < singleThreadTime, #H
"Parallel stream should be faster than single-threaded");
assertTrue(concurrentTime < singleThreadTime,
"Concurrent approach should be faster than single-threaded");
assertTrue(optimizedTime < singleThreadTime,
"Optimized approach should be faster than single-threaded");
System.out.println("Performance improvement:");
System.out.println("Parallel: " + String.format("%.2f", (double)
singleThreadTime / parallelTime) + "x faster");
System.out.println("Concurrent: " + String.format("%.2f", (double)
singleThreadTime / concurrentTime) + "x faster");
System.out.println("Optimized: " + String.format("%.2f", (double)
singleThreadTime / optimizedTime) + "x faster");
}Inspect the generated test’s account count, warm-up call, and timed comparisons before relying on its results. section 7.1.2
A dedicated harness such as JMH provides an alternative to hand-written timing logic. The JMH version makes warm-up, measurement iterations, forks, thread count, and account-count parameters explicit. Those settings still need to reflect the workload and the question being tested. section 7.1.2
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Benchmark)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
@Fork(1)
@Threads(1)
public class AccountFinderJMHBenchmark {
@Param({"1000", "10000", "100000"})
private int accountCount;
private AccountFinder accountFinder;
private List<Account> accounts;
private List<Integer> searchIds;
private Random random;
@Setup(Level.Trial)
public void setupTrial() {
random = new Random(42); #A
accounts = generateAccounts(accountCount);
accountFinder = new AccountFinder(accounts);
searchIds = generateRandomIds(random, 100);
}Review the JMH configuration and the account sizes selected for comparison. section 7.1.2
Custom measurement can also create maintenance costs outside a microbenchmark. A generated tracing solution added collectors, controllers, analysis code, and a separate simulation rather than reusing existing facilities. Compare traced and untraced latency, and consider established metrics libraries and parameterized load tests before adding more instrumentation. section 7.3.1
Run measurements before changing production code, interpret the results, and focus optimization on the important workload. More metrics can increase analysis effort and obscure the signals that matter. section 7.5
