liveBook · Manning

Memory traffic

As explained in Grokking GPU Programming

Memory traffic is the amount of data transferred between memory and processing units during computation.

Memory traffic is the amount of data transferred between memory and processing units during computation. Repeatedly loading the same data adds traffic even though the amount of unique data stays the same. If computation must wait for those loads, the GPU can spend time idle. section 12.1 section 12.8

Repeated loads in matrix multiplication

Multiply A (1024 × 1024) by B (1024 × 512), using four-byte floats. A naive GPU kernel assigns one output element to each thread. Each thread loads 1,024 values from a row of A and 1,024 from a column of B: 8,192 bytes for 1,024 multiply-adds. Counting a multiply-add as two operations gives 0.25 operations per byte loaded. section 12.1

Threads computing the same output row need the same values from A; threads computing the same output column need the same values from B. Across the calculation, each A element is requested 512 times and each B element 1,024 times. The roughly 6 MB of unique input data therefore generates about 4 GB of requested global memory traffic. These figures count input loads, not output writes. section 12.8

Tiling reduces repeated global memory requests. Threads first load a tile of each matrix into shared memory, then reuse those values for their calculations. With 32 × 32 tiles in this example, requested input traffic falls from about 4 GB to 128 MB. Threads synchronize after loading so all values are ready, and again after computing so the next load does not overwrite values still in use. section 12.8

Table 12.1 Memory traffic for A (1024×1024) × B (1024×512) matrix multiplication: tiled vs. naive
Metric
Tiled (TILE_SIZE = 32)
Naive (no tiling)
Each element requested from global memory
A: 512 / 32 = 16 times; B: 1,024 / 32 = 32 times
A: 512 times; B: 1,024 times
Total requested global memory traffic
128 MB (32× reduction)
4 GB
Inner loop reads
Shared memory (~20–30 cycles)
Global memory (~400–600 cycles)

Compare the requested loads and inner-loop memory sources for the naive and tiled kernels. section 12.8

For this tiled approach, the reduction in requested traffic is a factor of the tile size when all matrix dimensions are multiples of that size; otherwise, it is approximate. Count repeated requests, not just unique input bytes, when estimating traffic. section 12.8

How Grokking GPU Programming covers it

How deep it goes
Calculates per-thread loads and compares requested global traffic for naive and tiled kernels, including the aligned-dimension reduction rule.
Examples
Compares 4 GB of naive requested input traffic with 128 MB using 32 × 32 tiles for 1024 × 1024 and 1024 × 512 matrices.
What you can do after
Evaluate requested input traffic and arithmetic intensity for naive matrix multiplication, and estimate the reduction from tiling.
How it connects
Connects memory traffic to arithmetic intensity, matrix multiplication, tiling, shared memory, data reuse, and thread synchronization.

Also explained in

Compare how 2 books explain memory traffic →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.