liveBook · Manning

Shared memory

As explained in Grokking GPU Programming

Shared memory is a fast, explicitly managed memory space accessible to threads within a GPU thread block.

When GPU threads repeatedly need the same values, fetching each value from global memory can waste time. Shared memory is a small, fast, programmer-managed space where threads in one block can keep and share working data. Think of it as a classroom whiteboard: every thread in the block can use it, but other blocks have their own whiteboards. section 11.2 section 5.9

Unlike shared memory, registers are private to individual threads, while global memory is accessible across blocks. Shared memory lives on a streaming multiprocessor (SM) and is reclaimed when its block finishes; its contents do not carry over to another block or kernel launch. section 5.9 section 11.2

In CUDA, declare a fixed-size array with __shared__, or use extern __shared__ for a size supplied in bytes at kernel launch. Dynamic shared-memory declarations alias the same allocation, so separate regions must be laid out deliberately. section 5.11 section 11.14

A common pattern has three steps:

  • Load: threads cooperatively copy the needed values from global memory into shared memory. section 11.3
  • Collaborate: threads reuse or combine the staged values, synchronizing when one thread depends on another. section 11.3 section 11.4
  • Write back: threads put the result in global memory, where it can be used beyond the block. section 11.3

A reduction with a partial block

To sum six values with four threads per block, the second block has only two valid inputs. Its threads stage [data[4], data[5], 0, 0]: the extra threads write zero, the identity for addition. After a block-wide barrier, threads combine the staged values in rounds, with a barrier between rounds. The block can then contribute its partial sum to a global result. section 5.11 section 11.3

__shared__ int sdata[256];    #A
int tid = threadIdx.x;
int idx = blockIdx.x * blockDim.x + threadIdx.x;
sdata[tid] = (idx < n) ? data[idx] : 0;
__syncthreads();   #B

See the reduction load: each thread writes its own shared slot, including zero for an out-of-bounds input, before the block synchronizes. section 5.11

Shared memory starts with indeterminate contents. Write every slot before reading it; if another thread will read the slot, use __syncthreads() to separate the write from the read. Also synchronize before overwriting slots other threads may still be reading. Every thread in the block must reach a block-wide barrier, so do not put it inside a divergent branch. The barrier does not synchronize different blocks. section 11.13 section 11.4 section 11.14

Reusing matrix tiles

In tiled matrix multiplication, a block cooperatively loads chunks of two input matrices into shared memory. Its threads reuse those values to compute an output tile, then synchronize before loading the next chunks. For 1,024×1,024 and 1,024×512 inputs, 32×32 tiles reduce requested global-memory traffic from 4 GB to 128 MB in the illustrated comparison; actual benefit depends on the workload. section 12.2 section 12.8

Use shared memory when threads can reuse data, communicate, or improve an access pattern such as a transpose. A simple element-wise operation may gain nothing and incur extra overhead. The space is limited: exceeding the per-block limit can make a launch fail, while using a large legal allocation can reduce the number of blocks resident on an SM. Check device limits and call cudaGetLastError() after launches. Access patterns matter too: bank conflicts can slow shared-memory access, and padding can fix some column-access conflicts. section 11.14 section 11.12 section 11.0 section 11.6

How Grokking GPU Programming covers it

How deep it goes
Explains declarations, synchronization hazards, capacity limits, bank conflicts, and tiled kernels, with worked implementations and common mistakes.
Examples
Traces a padded reduction and compares global-memory traffic for naive and tiled matrix multiplication.
What it assumes
Introduces shared memory through a classroom analogy while using thread blocks, global indexing, and global memory in worked kernels.
What you can do after
Apply it via CUDA to stage data, coordinate block threads, and diagnose initialization, synchronization, and capacity mistakes.
How it connects
Connects shared memory to the GPU memory hierarchy, reduction, scan, histograms, tiling, synchronization, and occupancy.

Also explained in

Compare how 5 books explain shared memory →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.