liveBook · Manning

Custom benchmarking code

As explained in Vibe Engineering

Custom benchmarking code is application-specific measurement logic written outside a dedicated benchmarking harness.

Custom benchmarking code measures application performance without a dedicated benchmarking harness. It can produce useful comparisons, but plausible timings are not necessarily reliable: inadequate warm-up and JVM optimizations can distort results, while unrealistic input sizes can make the benchmark answer the wrong question. section 7.1.2

Account lookup comparison

An AI-generated AccountFinderPerformanceTest compared single-threaded and concurrent lookup strategies using 100,000 accounts without establishing that this matched production. Reducing the count to 10,000 better reflected the expected workload. The custom results favored the single-threaded version, but measurement uncertainty remained; one concurrent implementation also called indexOf inside a grouping operation, introducing quadratic work. section 7.1.2

Cursor
public class AccountFinderPerformanceTest {
    private AccountFinder accountFinder;
    private List<Account> accounts;
    private static final int ACCOUNT_COUNT = 100_000; #A
    private static final int SEARCH_COUNT = 1000;
 
    @BeforeEach
    void setUp() {
        accounts = generateAccounts(ACCOUNT_COUNT);
        accountFinder = new AccountFinder(accounts);
    }
 
    @Test
    void testPerformanceComparison() {
        Random random = new Random();
        List<Integer> searchIds = generateRandomIds(random, SEARCH_COUNT);
        warmUp(); #B
        long singleThreadTime = measureSingleThreadedPerformance(searchIds);
#C
        System.out.println("Single-threaded time: " + singleThreadTime + " ms");
        long parallelTime = measureParallelPerformance(searchIds); #D
        System.out.println("Parallel stream time: " + parallelTime + " ms");
        long concurrentTime = measureConcurrentPerformance(searchIds); #E
        System.out.println("Concurrent time: " + concurrentTime + " ms");
        long optimizedTime = measureOptimizedPerformance(searchIds); #F
        System.out.println("Optimized time: " + optimizedTime + " ms");
        verifyResultsConsistency(searchIds); #G
        assertTrue(parallelTime < singleThreadTime, #H
                "Parallel stream should be faster than single-threaded");
        assertTrue(concurrentTime < singleThreadTime,
                "Concurrent approach should be faster than single-threaded");
        assertTrue(optimizedTime < singleThreadTime,
                "Optimized approach should be faster than single-threaded");
 
        System.out.println("Performance improvement:");
        System.out.println("Parallel: " + String.format("%.2f", (double)
singleThreadTime / parallelTime) + "x faster");
        System.out.println("Concurrent: " + String.format("%.2f", (double)
singleThreadTime / concurrentTime) + "x faster");
        System.out.println("Optimized: " + String.format("%.2f", (double)
singleThreadTime / optimizedTime) + "x faster");
    }

Inspect the generated test’s account count, warm-up call, and timed comparisons before relying on its results. section 7.1.2

A dedicated harness such as JMH provides an alternative to hand-written timing logic. The JMH version makes warm-up, measurement iterations, forks, thread count, and account-count parameters explicit. Those settings still need to reflect the workload and the question being tested. section 7.1.2

Cursor code output part 1
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Benchmark)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
@Fork(1)
@Threads(1)
public class AccountFinderJMHBenchmark {
    @Param({"1000", "10000", "100000"})
    private int accountCount;
    private AccountFinder accountFinder;
    private List<Account> accounts;
    private List<Integer> searchIds;
    private Random random;
    @Setup(Level.Trial)
    public void setupTrial() {
        random = new Random(42); #A
        accounts = generateAccounts(accountCount);
        accountFinder = new AccountFinder(accounts);
        searchIds = generateRandomIds(random, 100);
    }

Review the JMH configuration and the account sizes selected for comparison. section 7.1.2

Custom measurement can also create maintenance costs outside a microbenchmark. A generated tracing solution added collectors, controllers, analysis code, and a separate simulation rather than reusing existing facilities. Compare traced and untraced latency, and consider established metrics libraries and parameterized load tests before adding more instrumentation. section 7.3.1

Run measurements before changing production code, interpret the results, and focus optimization on the important workload. More metrics can increase analysis effort and obscure the signals that matter. section 7.5

How Vibe Engineering covers it

How deep it goes
Examines generated timing code, workload sizing, JVM measurement hazards, and a JMH alternative without establishing that custom timings are definitive.
Examples
Compares account lookup strategies at different account counts and examines generated tracing for a word-search service.
What you can do after
Evaluate and understand benchmark results by checking workload assumptions, measurement hazards, and implementation costs before acting on timings.
How it connects
Connects benchmarking with JIT optimization, warm-up, algorithmic complexity, concurrency, and performance instrumentation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.