A GPU kernel can spend more time waiting for data than performing arithmetic. Arithmetic intensity measures how much floating-point work it performs per byte transferred to or from global memory: I = FLOPs / bytes. A higher ratio means more computation is done for the same amount of global-memory traffic. section 3.1.3 section 3.1.4
Compare a kernel’s intensity with the hardware’s ridge point: peak compute throughput divided by peak memory bandwidth. Below that point, the bandwidth roof limits the ideal throughput; above it, the compute roof limits it. This is a first-order model, not a guarantee of achieved performance: inefficient memory access or other overheads can leave a kernel below either roof. section 3.1.4 section 3.1.5
Naive FP32 matrix multiplication
For one output element, a scalar-output kernel reads K values from each input and writes one result. It performs 2K FLOPs and transfers (2K + 1) × 4 bytes, giving roughly 0.25 FLOP/byte for large K. Using the H200 SXM’s listed 4.8 TB/s bandwidth gives a calculated bandwidth ceiling of 1.2 TFLOP/s, well below its 67 TFLOP/s FP32 peak. This is a theoretical estimate, not a measured result. section 5.5.3
Figure 5.11 The theoretical roofline position of the naive FP32 matmul kernel on an NVIDIA H200 SXM. The 0.25 FLOP/byte arithmetic intensity intersects the listed 4.8 TB/s memory-bandwidth roof at 1.2 TFLOP/s. The FP32 ridge is approximately 14 FLOP/byte. Both positions are calculated from published specifications and are not measured kernel results.
The roofline plot places the naive FP32 matrix multiplication estimate against the H200’s memory and compute ceilings. section 5.5.3
Tiling raises intensity through reuse. Instead of assigning one output scalar to each program, assign a rectangle of outputs: a loaded value from A can contribute to several output columns, while a loaded value from B can contribute to several output rows. Each program accumulates its output tile in registers, sweeping through K with tl.dot before writing the tile to global memory. section 5.5.4
Fusion raises intensity by eliminating intermediate global-memory transfers. For SiLU followed by multiplication, separate FP32 kernels transfer 20 bytes per element: two input loads, an intermediate store and reload, and an output store. A fused kernel keeps the intermediate in registers and transfers 12 bytes per element. The traffic reduction gives a theoretical 1.67× speedup when memory traffic is the governing cost, not a promise of measured speedup. section 3.3 section 3.7
Listing 3..2 Fused SwiGLU kernel in Triton
@triton.jit
def swiglu(input_ptr, output_ptr, N, BLOCK_SIZE: tl.constexpr):
pid = tl.program_id(0) #1
H = N // 2 #2
offs = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE) #3
mask = offs < H
x1 = tl.load(input_ptr + offs, mask=mask, other=0.0) #4
x2 = tl.load(input_ptr + H + offs, mask=mask, other=0.0) #5
silu = x1 * tl.sigmoid(x1) #6
out = silu * x2 #7
tl.store(output_ptr + offs, out, mask=mask) #8
The fused SwiGLU kernel loads two contiguous input halves, computes SiLU and multiplication, and stores only the final result; masks protect incomplete blocks. section 3.3.1



