Memory traffic is the amount of data transferred between memory and processing units during computation. Repeatedly loading the same data adds traffic even though the amount of unique data stays the same. If computation must wait for those loads, the GPU can spend time idle. section 12.1 section 12.8
Repeated loads in matrix multiplication
Multiply A (1024 × 1024) by B (1024 × 512), using four-byte floats. A naive GPU kernel assigns one output element to each thread. Each thread loads 1,024 values from a row of A and 1,024 from a column of B: 8,192 bytes for 1,024 multiply-adds. Counting a multiply-add as two operations gives 0.25 operations per byte loaded. section 12.1
Threads computing the same output row need the same values from A; threads computing the same output column need the same values from B. Across the calculation, each A element is requested 512 times and each B element 1,024 times. The roughly 6 MB of unique input data therefore generates about 4 GB of requested global memory traffic. These figures count input loads, not output writes. section 12.8
Tiling reduces repeated global memory requests. Threads first load a tile of each matrix into shared memory, then reuse those values for their calculations. With 32 × 32 tiles in this example, requested input traffic falls from about 4 GB to 128 MB. Threads synchronize after loading so all values are ready, and again after computing so the next load does not overwrite values still in use. section 12.8
Table 12.1 Memory traffic for A (1024×1024) × B (1024×512) matrix multiplication: tiled vs. naive
|
Metric
|
Tiled
(TILE_SIZE = 32)
|
Naive (no
tiling)
|
|
Each
element requested from global memory
|
A: 512 / 32
= 16 times; B: 1,024 / 32 = 32 times
|
A: 512
times; B: 1,024 times
|
|
Total
requested global memory traffic
|
128 MB (32×
reduction)
|
4 GB
|
|
Inner loop
reads
|
Shared
memory (~20–30 cycles)
|
Global
memory (~400–600 cycles)
|
Compare the requested loads and inner-loop memory sources for the naive and tiled kernels. section 12.8
For this tiled approach, the reduction in requested traffic is a factor of the tile size when all matrix dimensions are multiples of that size; otherwise, it is approximate. Count repeated requests, not just unique input bytes, when estimating traffic. section 12.8

