liveBook · Manning

Data reuse

As explained in Grokking GPU Programming

Data reuse is the practice of retaining or sharing a value so that multiple computations can use it without repeatedly fetching it from its original storage.

If several computations need the same value, fetching it from global memory each time can mean many trips to slow storage. Data reuse avoids those repeated trips by retaining a value where it can be read again. On a GPU, threads in a block can load values into shared memory and read them many times, like students copying files onto a shared whiteboard. section 12.2 section 11.2

Ask first: will the same data be used repeatedly by one thread or by several threads? If not, shared memory may add synchronization overhead without a reuse benefit. An exception is using it to improve an access pattern, such as a transpose. Shared memory belongs to one block and lasts only until that block finishes; it is also a limited resource, so larger allocations can reduce how many blocks fit on a GPU multiprocessor. section 11.14 section 11.2

Tiled matrix multiplication

In a naive multiplication, threads repeatedly fetch matrix elements from global memory. Tiling lets a block cooperatively load chunks of both input matrices into shared memory, reuse those values for its output tile, and then move to the next chunks. For A of 1024×1024 and B of 1024×512, a tile size of 32 reduces requested global memory traffic from about 4 GB to 128 MB in this comparison. That is 32 times less requested traffic, not a guarantee of 32 times faster execution. section 12.2 section 12.8

Figure 12.6 The big picture of tiling. C is partitioned into T×T tiles, one per thread block; Block(m,n) owns the highlighted tile. To compute it, the block walks tile-row m of A and tile-column n of B, loading one T×T chunk of each into shared memory, computing on the pair, and moving on to the next chunk.

Follow one output tile as its block loads and uses successive chunks of the two input matrices. section 12.2

Keep reused data correct before counting the savings:

  • Write every shared-memory slot before reading it. Uninitialized slots can produce intermittent garbage; initialize them and synchronize when other threads will read them. section 11.13
  • In a tiled loop, synchronize after loading so readers see completed tiles, and again after computing so the next load does not overwrite values still in use. A block-wide barrier must not be placed in a divergent branch and does not synchronize different blocks. section 12.8 section 11.14
  • Match the loading scheme to the tile size. In a one-element-per-thread kernel, a smaller block leaves slots unfilled, while a larger block can corrupt the tile or write out of bounds. section 12.4

Overlapping stencil neighborhoods offer another reuse opportunity: a block can load its input stretch and surrounding halo cells once, then serve repeated reads from shared memory. Reuse is worthwhile when the saved fetches outweigh the extra storage and coordination. section 9.14 section 11.14

How Grokking GPU Programming covers it

How deep it goes
Works through tiled loading, synchronization, traffic counts, and resource tradeoffs; identifies stencil reuse without implementing it.
Examples
Compares naive and tiled matrix multiplication and identifies reuse in overlapping stencil neighborhoods.
What you can do after
Apply it using CUDA shared memory to cooperatively load and reuse matrix tiles while synchronizing block threads.
How it connects
Connects shared memory and cooperative loading with tiling, matrix multiplication, synchronization, and stencils.

Also explained in

Compare how 2 books explain data reuse →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.