A benchmark is a specified test or measurement procedure for evaluating performance. For a GPU kernel, the procedure must define what is timed: the CPU launches work asynchronously, so a host timer around a launch alone may not capture the GPU’s execution time. section 2.2.10
A practical kernel benchmark follows a few essential steps:
- Check correctness against a trusted result before timing. For floating-point outputs, triton.testing.assert_close allows absolute and relative tolerances because different accumulation orders can produce small rounding differences. section 2.2.10
- Decide whether to include compilation, allocation, or tuning. For steady-state kernel execution, warm up first: the initial Triton call compiles the kernel, and a new autotuning key can trigger configuration measurements. section 2.2.10 section 5.8.3
- Account for asynchronous GPU work. Synchronize around a host-timed interval, or use GPU events before and after the kernel to measure elapsed time in the GPU stream. section 2.2.10
- Repeat measurements and report a useful statistic, workload, and environment. triton.testing.do_bench provides a timing interface; comparisons can report medians and throughput alongside the hardware, software versions, shapes, and dtypes. section 2.2.10 section 5.9
Listing 2.2 Benchmarking code
import time
@triton.testing.perf_report( #1
triton.testing.Benchmark(
x_names=['size'], #2
x_vals=[2**i for i in range(10, 25)],
x_log=True,
line_arg='provider', #3
line_vals=['python', 'torch', 'triton'],
line_names=['Python (CPU loop)', 'PyTorch (GPU)', 'Triton (GPU)'],
styles=[('red', '-'), ('blue', '-'), ('green', '-')],
ylabel='GB/s',
plot_name='vector_addition_performance',
args={},
)
)
def benchmark(size, provider):
quantiles = [0.5, 0.2, 0.8]
x_gpu = torch.rand(size, device='cuda', dtype=torch.float32)
y_gpu = torch.rand_like(x_gpu)
if provider == 'python':
x_cpu = x_gpu.cpu().tolist()
y_cpu = y_gpu.cpu().tolist()
def python_add():
return [a + b for a, b in zip(x_cpu, y_cpu)]
for _ in range(5):
python_add()
n_reps = 20
times_ms = []
for _ in range(n_reps):
t0 = time.perf_counter()
python_add()
times_ms.append((time.perf_counter() - t0) * 1e3)
times_ms.sort()
ms = times_ms[n_reps // 2]
min_ms = times_ms[int(n_reps * 0.2)]
max_ms = times_ms[int(n_reps * 0.8)] #4
elif provider == 'torch':
ms, min_ms, max_ms = triton.testing.do_bench(lambda: x_gpu + y_gpu,
quantiles=quantiles)
elif provider == 'triton':
ms, min_ms, max_ms = triton.testing.do_bench(lambda: add(x_gpu, y_gpu),
quantiles=quantiles) #5
gbps = lambda ms: 12 * size / (ms * 1e-3) / 1e9 #6
return gbps(ms), gbps(min_ms), gbps(max_ms)
if __name__ == "__main__":
benchmark.run(print_data=True, show_plots=True,
save_path='./vec_add_perf') #7
The vector-addition benchmark configures a size sweep and compares Python, PyTorch, and Triton providers in a performance report. section 2.2.10
Choose a metric that fits the workload. The vector-addition comparison reports bandwidth in GB/s; the matrix-multiplication comparison reports throughput using an operation count of 2MNK. A roofline plot can relate measured throughput to hardware limits, but an intensity inferred from peak bandwidth is a lower bound, not necessarily the kernel’s actual operational intensity. section 2.2.10 section 5.9
Comparing matrix-multiplication kernels
For a 4096 × 4096 × 4096 FP16 product on one NVIDIA H200, the median across three benchmark processes rises from 0.415 TFLOP/s for a naive scalar kernel to 613.9 TFLOP/s for the optimized Triton kernel, versus 664.7 TFLOP/s for torch.matmul. Execution is measured after warm-up, excluding output allocation and JIT compilation. These results apply to the stated workload and environment, not every matrix shape or dtype. section 5.9
Table 5.4 Optimized Triton matrix multiplication on one NVIDIA H200. Times and rates are medians across 3 benchmark processes.
Shape (M,N,K) |
Dtype | p20 (ms) | p50 (ms) | p80 (ms) | Triton (TFLOP/s) | torch.matmul (TFLOP/s) |
Ratio |
|
FP16 |
0.013984 |
0.014304 |
0.014624 |
150.1 |
211.7 |
71.1% |
|
FP16 |
0.221696 |
0.223872 |
0.227808 |
613.9 |
664.7 |
92.3% |
|
FP16 |
0.053472 |
0.054144 |
0.054848 |
634.6 |
696.8 |
91.2% |
|
FP16 |
0.053216 |
0.053920 |
0.054592 |
637.2 |
695.9 |
91.3% |
|
BF16 |
0.071680 |
0.072480 |
0.073344 |
237.0 |
552.3 |
42.8% |
|
FP16 |
0.024704 |
0.024960 |
0.025408 |
62.75 |
55.68 |
112.7% |
The shape and dtype comparison shows why a single benchmark result should not be generalized: relative performance varies across the tested workloads. section 5.9
