liveBook · Manning

Benchmarking

As explained in GPU Programming with Triton

A benchmark is a specified test or measurement procedure used to evaluate program performance.

A benchmark is a specified test or measurement procedure for evaluating performance. For a GPU kernel, the procedure must define what is timed: the CPU launches work asynchronously, so a host timer around a launch alone may not capture the GPU’s execution time. section 2.2.10

A practical kernel benchmark follows a few essential steps:

  • Check correctness against a trusted result before timing. For floating-point outputs, triton.testing.assert_close allows absolute and relative tolerances because different accumulation orders can produce small rounding differences. section 2.2.10
  • Decide whether to include compilation, allocation, or tuning. For steady-state kernel execution, warm up first: the initial Triton call compiles the kernel, and a new autotuning key can trigger configuration measurements. section 2.2.10 section 5.8.3
  • Account for asynchronous GPU work. Synchronize around a host-timed interval, or use GPU events before and after the kernel to measure elapsed time in the GPU stream. section 2.2.10
  • Repeat measurements and report a useful statistic, workload, and environment. triton.testing.do_bench provides a timing interface; comparisons can report medians and throughput alongside the hardware, software versions, shapes, and dtypes. section 2.2.10 section 5.9
Listing 2.2 Benchmarking code
import time

@triton.testing.perf_report(              #1
    triton.testing.Benchmark(
        x_names=['size'],                 #2
        x_vals=[2**i for i in range(10, 25)],
        x_log=True,
        line_arg='provider',              #3
        line_vals=['python', 'torch', 'triton'],
        line_names=['Python (CPU loop)', 'PyTorch (GPU)', 'Triton (GPU)'],
        styles=[('red', '-'), ('blue', '-'), ('green', '-')],
        ylabel='GB/s',
        plot_name='vector_addition_performance',
        args={},
    )
)
def benchmark(size, provider):
    quantiles = [0.5, 0.2, 0.8]
    x_gpu = torch.rand(size, device='cuda', dtype=torch.float32)
    y_gpu = torch.rand_like(x_gpu)
    if provider == 'python':
        x_cpu = x_gpu.cpu().tolist()
        y_cpu = y_gpu.cpu().tolist()
        def python_add():
            return [a + b for a, b in zip(x_cpu, y_cpu)]
        for _ in range(5):
            python_add()
        n_reps = 20
        times_ms = []
        for _ in range(n_reps):
            t0 = time.perf_counter()
            python_add()
            times_ms.append((time.perf_counter() - t0) * 1e3)
        times_ms.sort()
        ms = times_ms[n_reps // 2]
        min_ms = times_ms[int(n_reps * 0.2)]
        max_ms = times_ms[int(n_reps * 0.8)]        #4
    elif provider == 'torch':
        ms, min_ms, max_ms = triton.testing.do_bench(lambda: x_gpu + y_gpu,
                                                       quantiles=quantiles)
    elif provider == 'triton':
        ms, min_ms, max_ms = triton.testing.do_bench(lambda: add(x_gpu, y_gpu),
                             quantiles=quantiles)   #5
    gbps = lambda ms: 12 * size / (ms * 1e-3) / 1e9 #6
    return gbps(ms), gbps(min_ms), gbps(max_ms)


if __name__ == "__main__":
    benchmark.run(print_data=True, show_plots=True,
                  save_path='./vec_add_perf') #7

The vector-addition benchmark configures a size sweep and compares Python, PyTorch, and Triton providers in a performance report. section 2.2.10

Choose a metric that fits the workload. The vector-addition comparison reports bandwidth in GB/s; the matrix-multiplication comparison reports throughput using an operation count of 2MNK. A roofline plot can relate measured throughput to hardware limits, but an intensity inferred from peak bandwidth is a lower bound, not necessarily the kernel’s actual operational intensity. section 2.2.10 section 5.9

Comparing matrix-multiplication kernels

For a 4096 × 4096 × 4096 FP16 product on one NVIDIA H200, the median across three benchmark processes rises from 0.415 TFLOP/s for a naive scalar kernel to 613.9 TFLOP/s for the optimized Triton kernel, versus 664.7 TFLOP/s for torch.matmul. Execution is measured after warm-up, excluding output allocation and JIT compilation. These results apply to the stated workload and environment, not every matrix shape or dtype. section 5.9

Table 5.4 Optimized Triton matrix multiplication on one NVIDIA H200. Times and rates are medians across 3 benchmark processes.
Shape (M,N,K) Dtype p20 (ms) p50 (ms) p80 (ms) Triton (TFLOP/s) torch.matmul (TFLOP/s) Ratio

(1024,1024,1024)

FP16

0.013984

0.014304

0.014624

150.1

211.7

71.1%

(4096,4096,4096)

FP16

0.221696

0.223872

0.227808

613.9

664.7

92.3%

(4096,1024,4096)

FP16

0.053472

0.054144

0.054848

634.6

696.8

91.2%

(1024,4096,4096)

FP16

0.053216

0.053920

0.054592

637.2

695.9

91.3%

(1024,1024,8192)

BF16

0.071680

0.072480

0.073344

237.0

552.3

42.8%

(1001,1537,509)

FP16

0.024704

0.024960

0.025408

62.75

55.68

112.7%

The shape and dtype comparison shows why a single benchmark result should not be generalized: relative performance varies across the tested workloads. section 5.9

How GPU Programming with Triton covers it

How deep it goes
Details correctness checks, timing boundaries, warm-up, repeated measurements, throughput, and interpretation of workload-specific results.
Examples
Compares vector addition across Python, PyTorch, and Triton, and benchmarks a matrix-multiplication kernel progression against torch.matmul.
What it assumes
Assumes intermediate Python and introduces kernel benchmarking alongside a first Triton kernel.
What you can do after
Apply it via Triton’s testing tools to check outputs and measure kernel performance against reference implementations.
How it connects
Connects GPU synchronization, JIT compilation, autotuning, and roofline analysis to meaningful performance measurements.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.