Chapter 11: Shared memory
This introduction explains why shared memory can speed up GPU computation and identifies the common collaboration pattern used in earlier kernels.
After this chapter you will be able to
- Shared memory performance role. Explain how using shared memory instead of reading every value directly from global memory can make a kernel faster.
- Shared memory collaboration pattern. Describe the common shared memory pattern used in earlier algorithms, using an example from the chapter introduction.
- GPU memory hierarchy. Describe the relative speed, capacity, and control of registers, shared memory, caches, and global memory.
- Shared memory data reuse. Explain why loading a repeatedly used value into shared memory can make a kernel faster than repeatedly reading it from global memory.
- Shared memory scope and tradeoffs. Explain why shared memory cannot replace global memory for everything and what resource tradeoffs come with allocating more of it.
- Shared memory scope and lifetime. Explain where shared memory resides, which threads can access it, and when its allocation ends.
- Reuse and cooperative access. Explain how shared memory supports data reuse, cooperative loading, and communication, and when its overhead is worthwhile.
- Static and dynamic declarations. Distinguish static and dynamic shared memory declarations and describe how to size a dynamic allocation at launch.
- Dynamic array aliasing and layout. Explain why multiple dynamic shared array declarations alias and describe how to lay out distinct arrays safely in one allocation.
- Three-step shared memory recipe. Trace the three-step recipe through the block reduction example, and describe how it can vary for another algorithm.
- Barrier placement and cost. Explain why the reduction uses barriers after loading and between rounds, and why unnecessary barriers should be avoided.
- Incomplete block load handling. Explain how to load an input whose length is not a multiple of the block size without leaving shared-memory slots uninitialized.
- Producer consumer dependency. Explain why a shared-memory producer must finish before another thread consumes its value, using the reduction recipe as an example.
- Shared memory barrier placement. Describe where to place barriers for read-after-write and write-after-read hazards, and when no internal barrier is needed.
- Barrier participation and scope. Explain the participation and block-scope limits of __syncthreads() and how cross-block dependencies are handled.
- Data reuse opportunities. Describe the two kinds of data reuse that can justify shared-memory staging, using an example of each.
- Reuse test limitations. Apply the data-reuse test to a pure map and explain why the test is a guideline rather than a guarantee.
- Reduction and round barriers. Explain how reduction combines values in shared memory and where its barriers belong.
- Blelloch scan recipe. Trace the 3-Step Recipe through the Blelloch scan, including its phases and synchronization.
- Hillis-Steele ping-pong buffers. Explain why Hillis-Steele uses two shared-memory buffers during its doubling-stride rounds.
- Histogram recipe steps. Describe how the three steps apply to a block-local histogram.
- Shared memory access performance. Explain how an access pattern can affect shared memory performance through bank conflicts.
- Warp grouping in blocks. Explain how threads are grouped into warps in the section’s 32×32 block and why that grouping matters for bank conflicts.
- Shared memory bank mapping. Explain how shared-memory banks map 4-byte array elements and when accesses to them conflict.
- Stride and conflict degree. Apply the bank mapping to predict conflicts for the section’s power-of-two strides and explain the bandwidth effect on the access.
- Same address broadcast. Distinguish a bank conflict from threads reading the same shared-memory address, including broadcast and multicast.
- Padding and bank alignment. Explain how adding a spacer column changes row alignment and prevents column-access bank conflicts in the 4-bank example and the 32-bank tile.
- Tiled transpose data flow. Trace how the shared-memory transpose reads a tile, synchronizes, and writes it transposed with coalesced global accesses.
- Padding for column access. Explain why the transpose uses a 32×33 shared-memory tile instead of a plain 32×32 tile.
- Column access bank conflicts. Explain why a warp reading down a column of a [32][32] shared-memory tile can be slow.
- Shared memory resource limits. Describe how a block's shared-memory allocation can affect occupancy or kernel launch.
- Uninitialized shared memory reads. Explain the risk of reading a shared-memory slot that no thread has initialized.
- Column Access Bank Conflicts. Explain why column access to a [32][32] shared-memory tile can be slow and how padding the tile fixes it.
- Per-block limit and launch failure. Explain what happens when a kernel requests too much shared memory and how to detect and avoid the failure, including the distinction for oversized static arrays.
- Shared memory and occupancy. Explain how a large but legal shared memory allocation can reduce throughput by limiting the blocks and warps available on an SM.
- Shared memory initialization. Explain why shared memory must be written before it is read and how to make those writes available to other threads.
What it builds on
- The memory bottleneck (chapter 10) · The memory bottleneck can limit kernel speed.
- Where threads share data: shared memory (chapter 5) · Reduction threads share data in shared memory.
- Part 1: Setup with shared memory and the double buffer (chapter 6) · Hillis-Steele scan uses shared memory for working data.
- A faster fix: privatization in shared memory (chapter 8) · Histogram counts can be accumulated in shared memory.
- The three-level hierarchy: grids, blocks, and threads (chapter 3) · The organization of threads into blocks and blocks into a grid.
- How the hardware runs your threads: warps and SMs (chapter 2) · SMs and how hardware runs threads
- One thread, one element (chapter 4) · The one-thread-per-element pattern
- The launch syntax: starting those threads (chapter 2) · Kernel launch configuration syntax
Section by section
Introduction
This introduction explains why shared memory can speed up GPU computation and identifies the common collaboration pattern used in earlier kernels.
- Shared memory performance role: Explain how using shared memory instead of reading every value directly from global memory can make a kernel faster.
Why it matters: Where data lives during computation can make a large difference to kernel performance. - Shared memory collaboration pattern: Describe the common shared memory pattern used in earlier algorithms, using an example from the chapter introduction.
Why it matters: Recognizing this recurring pattern helps you apply shared memory deliberately in other kernels.
Key terms: shared memory, global memory, reduction, prefix scans, histogram
11.1 When your kernel is slow
This section explains the GPU memory hierarchy and why reusing data in shared memory can reduce latency, along with its scope and resource limits.
- GPU memory hierarchy: Describe the relative speed, capacity, and control of registers, shared memory, caches, and global memory.
Why it matters: The hierarchy helps explain where memory delays come from and which storage the programmer controls. - Shared memory data reuse: Explain why loading a repeatedly used value into shared memory can make a kernel faster than repeatedly reading it from global memory.
Why it matters: Reusing a locally stored value is the central reason shared memory can speed up a kernel. - Shared memory scope and tradeoffs: Explain why shared memory cannot replace global memory for everything and what resource tradeoffs come with allocating more of it.
Why it matters: Shared memory is useful only when its limited space, block scope, and resource costs fit the kernel. - Shared memory configuration limits: Describe how to check the GPU's per-block shared-memory limit and request more than the default allocation.
Why it matters: Checking the actual device limit prevents assuming every GPU supports the same allocation. - Conflict-free latency caveat: Explain what the conflict-free shared-memory latency means and how a bad access pattern can affect it.
Why it matters: The advertised latency is not guaranteed for every shared-memory access pattern. - Repeated read cycle comparison: Apply the section's cycle-counting example to compare 50 global reads with one global load followed by 49 shared-memory reads.
Why it matters: A simple cycle estimate shows how repeated reuse can turn a faster memory level into a substantial overall saving.
Key terms: registers, shared memory, L1 cache, L2 cache, global memory, DRAM, latency, L2 hit, block, SRAM
11.2 Introducing shared memory
Introduces shared memory’s block-local uses, lifetime, declaration methods, and safe allocation of multiple arrays.
- Shared memory scope and lifetime: Explain where shared memory resides, which threads can access it, and when its allocation ends.
Why it matters: The block boundary determines who can communicate through shared memory and how long it remains available. - Reuse and cooperative access: Explain how shared memory supports data reuse, cooperative loading, and communication, and when its overhead is worthwhile.
Why it matters: Shared memory is most useful when its sharing and reuse save enough work to justify using it. - Static and dynamic declarations: Distinguish static and dynamic shared memory declarations and describe how to size a dynamic allocation at launch.
Why it matters: Choosing the right declaration and byte count prevents incorrect or unavailable shared-memory allocations. - Dynamic array aliasing and layout: Explain why multiple dynamic shared array declarations alias and describe how to lay out distinct arrays safely in one allocation.
Why it matters: Correctly partitioning the single dynamic region avoids overlapping arrays and out-of-bounds or misaligned accesses. - Fixed size layout alternatives: Describe the simpler alternatives for laying out fixed-size shared arrays, including when a struct is useful.
Why it matters: For fixed sizes, compiler-managed layout can avoid error-prone manual pointer offsets.
Key terms: shared memory, SM, block, threads, data reuse, cooperative loading, thread communication, global memory, __shared__, extern __shared__
11.3 The 3-Step Recipe
This section explains the three-step shared-memory recipe through a block reduction, including synchronization, cooperative loading, and handling incomplete blocks.
- Three-step shared memory recipe: Trace the three-step recipe through the block reduction example, and describe how it can vary for another algorithm.
Why it matters: This recipe provides a reusable way to organize shared-memory algorithms. - Barrier placement and cost: Explain why the reduction uses barriers after loading and between rounds, and why unnecessary barriers should be avoided.
Why it matters: Correct barrier placement prevents threads from reading shared data before it is ready without adding needless waits. - Cooperative coalesced loading: Describe how threads cooperate to load the block's data and why their global-memory accesses coalesce in the example.
Why it matters: Cooperative, coalesced loading gets data into the shared workspace efficiently for subsequent reuse. - Incomplete block load handling: Explain how to load an input whose length is not a multiple of the block size without leaving shared-memory slots uninitialized.
Why it matters: Initializing every slot makes the reduction safe even when the final block extends beyond the input.
Key terms: 3-Step Recipe, global memory, shared memory, scratchpad memory, software-managed caching, __syncthreads(), barrier, block, cooperative loading, coalescing
11.4 Producers and consumers
This section explains producer-consumer synchronization in shared memory and when staging data there is worthwhile.
- Producer consumer dependency: Explain why a shared-memory producer must finish before another thread consumes its value, using the reduction recipe as an example.
Why it matters: This dependency explains why the reduction cannot safely proceed from one phase to the next without synchronization. - Shared memory barrier placement: Describe where to place barriers for read-after-write and write-after-read hazards, and when no internal barrier is needed.
Why it matters: Correct placement protects both readers and writers without adding barriers where they serve no purpose. - Barrier participation and scope: Explain the participation and block-scope limits of __syncthreads() and how cross-block dependencies are handled.
Why it matters: A block barrier cannot safely replace either full block participation or a kernel boundary between dependent blocks. - Data reuse opportunities: Describe the two kinds of data reuse that can justify shared-memory staging, using an example of each.
Why it matters: Recognizing reuse helps identify where cooperative or repeated on-chip access can repay the cost of staging. - Reuse test limitations: Apply the data-reuse test to a pure map and explain why the test is a guideline rather than a guarantee.
Why it matters: The reuse test guides optimization choices, but performance also depends on occupancy and memory access patterns.
Key terms: producer-consumer dependency, producer, consumer, __syncthreads(), read-after-write, write-after-read, divergent branch, kernel boundary, data-reuse test, shared memory
11.5 Applying the 3-Step Recipe
This section applies the shared-memory 3-Step Recipe to several algorithms and connects its data-reuse principle to production GPU techniques.
- Reduction and round barriers: Explain how reduction combines values in shared memory and where its barriers belong.
Why it matters: Reduction shows why synchronization depends on data dependencies, not merely on the boundaries of the three steps. - Blelloch scan recipe: Trace the 3-Step Recipe through the Blelloch scan, including its phases and synchronization.
Why it matters: This shows how the same load, collaborate, and write-back structure can produce all prefix sums rather than just one total. - Hillis-Steele ping-pong buffers: Explain why Hillis-Steele uses two shared-memory buffers during its doubling-stride rounds.
Why it matters: The two buffers preserve the values a round needs while threads produce the next set of values. - Histogram recipe steps: Describe how the three steps apply to a block-local histogram.
Why it matters: The histogram illustrates a different collaboration step: atomic bin updates instead of rounds of pairwise combination. - Tiled matrix element reuse: Describe the shared-memory reuse idea introduced for tiled matrix multiplication.
Why it matters: Reusing loaded elements is the shared-memory idea that the later matrix multiplication chapter develops. - Production data reuse mechanisms: Distinguish torch.compile kernel fusion from FlashAttention as examples of on-chip data reuse.
Why it matters: These examples show that avoiding expensive intermediate memory traffic can use different on-chip mechanisms.
Key terms: parallel reduction, stride-halving tree, __syncthreads(), barrier, Blelloch, up-sweep, seed step, down-sweep, prefix values, Hillis-Steele
11.6 Bank conflicts
This section identifies bank conflicts as a cause of shared memory slowdowns.
- Shared memory access performance: Explain how an access pattern can affect shared memory performance through bank conflicts.
Why it matters: Fast shared memory can still become a bottleneck if its access pattern causes conflicts.
Key terms: shared memory, access pattern, bank conflicts
11.7 Predicting bank conflicts from the access pattern
This section explains how warp membership and shared-memory access patterns determine bank conflicts and their effect on individual accesses.
- Warp grouping in blocks: Explain how threads are grouped into warps in the section’s 32×32 block and why that grouping matters for bank conflicts.
Why it matters: Identifying the threads in a warp tells you which simultaneous accesses can conflict. - Shared memory bank mapping: Explain how shared-memory banks map 4-byte array elements and when accesses to them conflict.
Why it matters: The bank mapping lets you turn a warp’s addresses into a prediction about conflicts. - Stride and conflict degree: Apply the bank mapping to predict conflicts for the section’s power-of-two strides and explain the bandwidth effect on the access.
Why it matters: Predicting conflicts from a stride helps explain why a particular shared-memory access is slow. - Same address broadcast: Distinguish a bank conflict from threads reading the same shared-memory address, including broadcast and multicast.
Why it matters: The same-bank test alone can overstate conflicts unless you account for shared-address reads.
Key terms: warp, threadIdx.x, threadIdx.y, row-major flattening, shared memory, banks, bank conflict, element index mod 32, stride, N-way bank conflict
11.8 Avoiding bank conflicts with padding
This section explains how an extra shared-memory column prevents column-access bank conflicts and weighs its memory cost against the benefit.
- Padding and bank alignment: Explain how adding a spacer column changes row alignment and prevents column-access bank conflicts in the 4-bank example and the 32-bank tile.
Why it matters: Padding can turn a conflicting column access into one that uses all the banks. - Padding cost and benefit: Describe the benefit and shared-memory cost of padding a 32-by-32 tile, including when the extra memory may be a concern.
Why it matters: The improvement is inexpensive in this example, but its memory cost can matter for occupancy.
Key terms: padding, row-major order, banks, spacer column, 32-way conflict, shared memory, occupancy
11.9 A first taste of tiling: transpose
This section uses a tiled matrix transpose to demonstrate coalesced global access, synchronization, and conflict-free shared-memory access.
- Tiled transpose data flow: Trace how the shared-memory transpose reads a tile, synchronizes, and writes it transposed with coalesced global accesses.
Why it matters: The tile allows the transpose to avoid the uncoalesced global access of a naive implementation. - Padding for column access: Explain why the transpose uses a 32×33 shared-memory tile instead of a plain 32×32 tile.
Why it matters: Coalescing global accesses is not enough if the shared-memory transpose creates a severe bank conflict. - Transpose boundary assumption: State the transpose kernel's size assumption and describe the bounds checks needed when it does not hold.
Why it matters: The simple example needs extra checks to handle matrix sizes that do not fill whole tiles. - Matrix tiling traffic benefit: Describe the broader matrix tiling idea and its stated benefit for global memory traffic.
Why it matters: The transpose is a small example of a technique that can reduce traffic in larger matrix operations. - Stride four bank conflict: Apply the given bank formula to identify the conflict when threads read data[threadIdx.x * 4].
Why it matters: Tracing the bank pattern lets you spot a conflict before running the code.
Key terms: shared memory tile, coalesced, __syncthreads(), threadIdx.x, threadIdx.y, bank conflict, padding, tile[32][32], tile[32][33], bounds checks
11.10 Common mistakes
This section identifies shared-memory mistakes that can cause slowdowns, invalid results, or launch failures.
- Column access bank conflicts: Explain why a warp reading down a column of a [32][32] shared-memory tile can be slow.
Why it matters: A shared-memory access pattern can undermine the speed it is meant to provide. - Shared memory resource limits: Describe how a block's shared-memory allocation can affect occupancy or kernel launch.
Why it matters: A shared-memory allocation can be too costly even before the kernel does useful work. - Uninitialized shared memory reads: Explain the risk of reading a shared-memory slot that no thread has initialized.
Why it matters: An uninitialized read can produce unreliable results that are not apparent on every run or GPU.
Key terms: warp, bank, shared memory, [32][32] tile, block, occupancy, initialized
11.11 Mistake 1: Bank conflicts in column access
This section explains how padding a shared-memory tile avoids bank conflicts during column access.
- Column Access Bank Conflicts: Explain why column access to a [32][32] shared-memory tile can be slow and how padding the tile fixes it.
Why it matters: A correct kernel can still be slow if its shared-memory accesses conflict.
Key terms: bank conflicts, shared memory, column access, padding, warp
11.12 Mistake 2: Exceeding shared memory limits
This section explains how shared memory limits can cause kernel launch failures or reduce performance, and how to check and stay within the applicable limit.
- Per-block limit and launch failure: Explain what happens when a kernel requests too much shared memory and how to detect and avoid the failure, including the distinction for oversized static arrays.
Why it matters: A failed launch can leave old output in place and be mistaken for a successful computation. - Shared memory and occupancy: Explain how a large but legal shared memory allocation can reduce throughput by limiting the blocks and warps available on an SM.
Why it matters: A kernel can fit within the limit and still run slowly because its shared memory use restricts concurrency.
Key terms: shared memory per block, cudaGetLastError(), __shared__, prop.sharedMemPerBlock, prop.sharedMemPerBlockOptin, cudaFuncSetAttribute, shared memory, SM, blocks, warps
11.13 Mistake 3: Reading uninitialized shared memory
This section explains why reading uninitialized shared memory causes unreliable results and how to prevent it.
- Shared memory initialization: Explain why shared memory must be written before it is read and how to make those writes available to other threads.
Why it matters: Initializing shared memory before use prevents hard-to-reproduce garbage results.
Key terms: __shared__ memory, indeterminate, __syncthreads(), zero-initialized
11.14 Summary
Where this is used later
- The solution: tiled matrix multiplication (chapter 12) · Uses shared memory in a tiled matrix multiplication solution.
- Constant memory within the hierarchy (chapter 13) · Provides context for positioning another memory type in the hierarchy.
- What limits occupancy: three resources (chapter 14) · Builds on the effect of per-block resource use on simultaneous blocks.
- Step 1: Load tiles into shared memory (chapter 12) · Uses block-local shared memory to hold matrix tiles.
- (chapter intro) (chapter 12) · Shared memory as storage accessible near threads.
- Understanding the problem (chapter 12) · The memory hierarchy and caches mentioned in the comparison.
- Walkthrough of a tiled kernel (chapter 12) · Uses shared-memory loading and computation in a tiled kernel.
- Step 2: Synchronize (chapter 12) · Uses synchronization between tile loading and computation.
Concepts explained in this chapter
- shared memoryShared memory is a fast, explicitly managed memory space accessible to threads within a GPU thread block.
- data-reuse testThe data-reuse test is a criterion for deciding whether a kernel benefits from shared memory based on whether data is read repeatedly.
- memory hierarchyA memory hierarchy is an arrangement of storage levels with differing capacities, access latencies, and bandwidths.
- data reuseData reuse is the practice of retaining or sharing a value so that multiple computations can use it without repeatedly fetching it from its original storage.
- shared memory bank conflictA bank conflict occurs when threads in the same GPU warp access different data assigned to the same shared-memory bank, requiring serialized service.
- three-step recipeThe three-step recipe is a shared-memory workflow of loading data, collaborating on it within a block, and writing results back.
- __syncthreads()__syncthreads is a CUDA barrier operation that makes threads in one block wait until all threads in that block reach the barrier.
- strideMemory stride is the distance between memory locations accessed by successive elements of an access pattern.
- paddingPadding adds extra elements to a data structure, commonly with neutral values, to meet an algorithm's size requirements.
- kernel boundaryA kernel boundary is the transition between GPU kernel executions that, in the described ordinary same-stream launch sequence, provides a grid-wide synchronization point and makes prior writes visible.
- producer-consumer dependencyA dependency in which a consumer's read of shared data must follow the producer's write to that data.
- read-after-write hazardA read-after-write hazard occurs when a read may happen before a preceding write to the same location is complete.
- write-after-read raceA write-after-read hazard occurs when a write may overwrite a location before a preceding read from it is complete.
- memory coalescingMemory coalescing combines memory requests from GPU threads into fewer transactions when their accesses are suitably grouped.
- tileA tile is a smaller rectangular block of elements taken from a larger matrix or other data structure.
- cooperative loadingCooperative loading is a technique in which multiple threads work together to load data into a shared storage area.
- dynamic shared memoryDynamic shared memory is shared memory whose capacity is specified when a CUDA kernel is launched.
- static shared memoryStatic shared memory is shared memory whose capacity is fixed in a CUDA kernel declaration.
- thread communicationThread communication is the exchange of data between threads during concurrent execution.
- shared memory broadcastA memory broadcast supplies one value read from a single address to multiple threads.
- synchronizationSynchronization is the coordination of concurrent computations so that their execution observes required ordering or completion constraints.
- row-major orderRow-major order is a multidimensional array layout in which elements of each row are contiguous in memory.
- L2 cacheAn L2 cache is a hardware-managed cache that stores copies of data to reduce the cost of memory access.
- memory bankA memory bank is an independently accessed division of memory that can serve requests concurrently with other banks.
- alignmentMemory alignment is the placement of data at an address satisfying the boundary requirements of its type or hardware access.
- tilingTiling is a technique that partitions data or computation into smaller blocks to enable reuse and reduce memory traffic.
- double bufferingDouble buffering uses two buffers alternately to separate data being read from data being written.
- identity elementAn identity element is a value that leaves the result of an operation unchanged when combined with another value.
- cudaGetLastErrorcudaGetLastError is a CUDA runtime API function that returns the latest CUDA error status for the calling host thread.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
