When GPU threads repeatedly need the same values, fetching each value from global memory can waste time. Shared memory is a small, fast, programmer-managed space where threads in one block can keep and share working data. Think of it as a classroom whiteboard: every thread in the block can use it, but other blocks have their own whiteboards. section 11.2 section 5.9
Unlike shared memory, registers are private to individual threads, while global memory is accessible across blocks. Shared memory lives on a streaming multiprocessor (SM) and is reclaimed when its block finishes; its contents do not carry over to another block or kernel launch. section 5.9 section 11.2
In CUDA, declare a fixed-size array with __shared__, or use extern __shared__ for a size supplied in bytes at kernel launch. Dynamic shared-memory declarations alias the same allocation, so separate regions must be laid out deliberately. section 5.11 section 11.14
A common pattern has three steps:
- Load: threads cooperatively copy the needed values from global memory into shared memory. section 11.3
- Collaborate: threads reuse or combine the staged values, synchronizing when one thread depends on another. section 11.3 section 11.4
- Write back: threads put the result in global memory, where it can be used beyond the block. section 11.3
A reduction with a partial block
To sum six values with four threads per block, the second block has only two valid inputs. Its threads stage [data[4], data[5], 0, 0]: the extra threads write zero, the identity for addition. After a block-wide barrier, threads combine the staged values in rounds, with a barrier between rounds. The block can then contribute its partial sum to a global result. section 5.11 section 11.3
__shared__ int sdata[256]; #A int tid = threadIdx.x; int idx = blockIdx.x * blockDim.x + threadIdx.x; sdata[tid] = (idx < n) ? data[idx] : 0; __syncthreads(); #B
See the reduction load: each thread writes its own shared slot, including zero for an out-of-bounds input, before the block synchronizes. section 5.11
Shared memory starts with indeterminate contents. Write every slot before reading it; if another thread will read the slot, use __syncthreads() to separate the write from the read. Also synchronize before overwriting slots other threads may still be reading. Every thread in the block must reach a block-wide barrier, so do not put it inside a divergent branch. The barrier does not synchronize different blocks. section 11.13 section 11.4 section 11.14
Reusing matrix tiles
In tiled matrix multiplication, a block cooperatively loads chunks of two input matrices into shared memory. Its threads reuse those values to compute an output tile, then synchronize before loading the next chunks. For 1,024×1,024 and 1,024×512 inputs, 32×32 tiles reduce requested global-memory traffic from 4 GB to 128 MB in the illustrated comparison; actual benefit depends on the workload. section 12.2 section 12.8
Use shared memory when threads can reuse data, communicate, or improve an access pattern such as a transpose. A simple element-wise operation may gain nothing and incur extra overhead. The space is limited: exceeding the per-block limit can make a launch fail, while using a large legal allocation can reduce the number of blocks resident on an SM. Check device limits and call cudaGetLastError() after launches. Access patterns matter too: bank conflicts can slow shared-memory access, and padding can fix some column-access conflicts. section 11.14 section 11.12 section 11.0 section 11.6




