liveBook · Manning

Arithmetic intensity

As explained in GPU Programming with Triton

Arithmetic intensity is the ratio of floating-point operations performed to bytes transferred between a kernel and global memory.

A GPU kernel can spend more time waiting for data than performing arithmetic. Arithmetic intensity measures how much floating-point work it performs per byte transferred to or from global memory: I = FLOPs / bytes. A higher ratio means more computation is done for the same amount of global-memory traffic. section 3.1.3 section 3.1.4

Compare a kernel’s intensity with the hardware’s ridge point: peak compute throughput divided by peak memory bandwidth. Below that point, the bandwidth roof limits the ideal throughput; above it, the compute roof limits it. This is a first-order model, not a guarantee of achieved performance: inefficient memory access or other overheads can leave a kernel below either roof. section 3.1.4 section 3.1.5

Naive FP32 matrix multiplication

For one output element, a scalar-output kernel reads K values from each input and writes one result. It performs 2K FLOPs and transfers (2K + 1) × 4 bytes, giving roughly 0.25 FLOP/byte for large K. Using the H200 SXM’s listed 4.8 TB/s bandwidth gives a calculated bandwidth ceiling of 1.2 TFLOP/s, well below its 67 TFLOP/s FP32 peak. This is a theoretical estimate, not a measured result. section 5.5.3

Figure 5.11 The theoretical roofline position of the naive FP32 matmul kernel on an NVIDIA H200 SXM. The 0.25 FLOP/byte arithmetic intensity intersects the listed 4.8 TB/s memory-bandwidth roof at 1.2 TFLOP/s. The FP32 ridge is approximately 14 FLOP/byte. Both positions are calculated from published specifications and are not measured kernel results.
CH05 F11 Fartale

The roofline plot places the naive FP32 matrix multiplication estimate against the H200’s memory and compute ceilings. section 5.5.3

Tiling raises intensity through reuse. Instead of assigning one output scalar to each program, assign a rectangle of outputs: a loaded value from A can contribute to several output columns, while a loaded value from B can contribute to several output rows. Each program accumulates its output tile in registers, sweeping through K with tl.dot before writing the tile to global memory. section 5.5.4

Fusion raises intensity by eliminating intermediate global-memory transfers. For SiLU followed by multiplication, separate FP32 kernels transfer 20 bytes per element: two input loads, an intermediate store and reload, and an output store. A fused kernel keeps the intermediate in registers and transfers 12 bytes per element. The traffic reduction gives a theoretical 1.67× speedup when memory traffic is the governing cost, not a promise of measured speedup. section 3.3 section 3.7

Listing 3..2 Fused SwiGLU kernel in Triton
@triton.jit
def swiglu(input_ptr, output_ptr, N, BLOCK_SIZE: tl.constexpr):
    pid = tl.program_id(0) #1

    H = N // 2 #2
    offs = pid * BLOCK_SIZE + tl.arange(0, BLOCK_SIZE) #3
    mask = offs < H

    x1 = tl.load(input_ptr + offs, mask=mask, other=0.0) #4
    x2 = tl.load(input_ptr + H + offs, mask=mask, other=0.0) #5

    silu = x1 * tl.sigmoid(x1) #6
    out = silu * x2 #7

    tl.store(output_ptr + offs, out, mask=mask) #8

The fused SwiGLU kernel loads two contiguous input halves, computes SiLU and multiplication, and stores only the final result; masks protect incomplete blocks. section 3.3.1

How GPU Programming with Triton covers it

How deep it goes
Derives per-element traffic and FLOP estimates, compares hardware ridge points, and relates the estimates to roofline ceilings and implementation choices.
Examples
Works through naive FP32 matrix multiplication and fused SiLU multiplication, with a Triton SwiGLU kernel.
What it assumes
Assumes familiarity with GPU kernel launches, pointer arithmetic, and masking from earlier kernels.
What you can do after
Diagnose memory-bound kernels with arithmetic-intensity estimates and evaluate fusion or tiling as ways to reduce traffic per operation.
How it connects
Connects the memory wall and roofline model to kernel fusion, matrix multiplication, tiling, and data reuse.

Also explained in

Compare how 4 books explain arithmetic intensity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.