liveBook · Manning

Cooperative loading

As explained in Grokking GPU Programming

Cooperative loading is a technique in which multiple threads work together to load data into a shared storage area.

In CUDA, threads in a block divide the work of copying a chunk from global memory into shared memory—like a bucket brigade. They can then reuse that data while collaborating, instead of each fetching the same values independently. section 11.2 section 11.3

Keep the loading phase correct:

  • Have threads load their assigned pieces, then synchronize before any thread reads data written by another. section 11.3
  • For a partial chunk, write an identity value into out-of-range slots; skipping the store leaves shared memory uninitialized. section 11.3

Loading for a reduction

With 256 threads, each thread loads one input value. Neighboring loads combine into 8 transactions instead of 256 separate fetches; the block then reduces the values in shared memory. section 11.3

Figure 11.3 The 3-Step Recipe, shown for a single block: load data from global memory into shared memory, collaborate there over several rounds, then write the result back to global memory. The worked example inside the panels is the parallel reduction from chapter 5. Time flows downward, and each solid arrow is one thread’s write. The curve marks the time when each thread executes the __syncthreads() statement, and the empty space below the curve depicts the time that each thread waits for all threads to complete. The horizontal line marks the time when the last thread executes __syncthreads() (the last thread to arrive never waits), after which all threads proceed, and the shared memory state below the line is the result. In later rounds of Step 2, only the threads still holding data are drawn; the remaining threads call __syncthreads() immediately and wait, so every barrier still involves the whole block. In Step 3, the dashed arrow is an atomicAdd: each block’s thread 0 atomically adds its partial sum into the single result variable in global memory.

Trace the load, collaboration, and write-back phases, including where threads wait at block-wide barriers. section 11.3

How Grokking GPU Programming covers it

How deep it goes
Explains per-thread loading, coalesced accesses, synchronization, bounds guards, and shared-memory limits.
Examples
Uses a 256-thread reduction load and shared tiles of matrices A and B.
What you can do after
Apply it via CUDA shared memory by assigning threads input pieces, synchronizing before shared reads, and guarding out-of-range loads.
How it connects
Connects shared memory with parallel reduction and tiled matrix multiplication.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.