Shared memory
Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.
Grokking GPU Programming
Shared memory is a fast, explicitly managed memory space accessible to threads within a GPU thread block.
- How it is used
- Uses shared memory for block-level data reuse, cooperative loading, and thread communication in parallel GPU algorithms.
- How deep it goes
- Explains declarations, synchronization hazards, capacity limits, bank conflicts, and tiled kernels, with worked implementations and common mistakes.
- Examples
- Traces a padded reduction and compares global-memory traffic for naive and tiled matrix multiplication.
- What you can do after
- Apply it via CUDA to stage data, coordinate block threads, and diagnose initialization, synchronization, and capacity mistakes.
CUDA for LLMs
Shared memory is a low-latency on-chip GPU memory region used for cooperation among threads.
- How it is used
- Uses block-local shared memory to reduce global-memory traffic and coordinate threads in deep-learning kernels.
- How deep it goes
- Explains allocation, cooperative loading, synchronization, reductions, tiling, and implicit tensor-core transfers in CUDA examples.
- Examples
- Works through GEMV reduction, tiled GEMM, fused LayerNorm, and Flash Attention tiles.
- What you can do after
- Build shared-memory CUDA kernels that cooperate on reductions and reuse data across tiled or fused computations.
GPU Programming with Triton
Shared memory is explicitly managed on-chip GPU memory visible to all threads in a single block.
- How it is used
- Motivates fast on-chip storage for thread cooperation and data reuse when global-memory access limits GPU performance.
- How deep it goes
- Explains staging, synchronization, access-pattern changes, and cache partitioning, while leaving Triton storage decisions to compiler lowering.
- Examples
- Uses matrix multiplication tiles and inter-warp partial-result exchange in reductions and scans.
- What you can do after
- Evaluate/understand when shared storage can reduce global-memory reads and how its capacity competes with L1 cache.
Read how GPU Programming with Triton explains shared memory →
Fast Python: High performance techniques for large datasets
- How it is used
- Uses managed shared memory for local process pipelines and array sharing to avoid unnecessary data copies.
- How deep it goes
- Demonstrates array sharing checks and a Plasma pipeline while deferring low-level shared-memory implementation.
- Examples
- Uses a flipped image view and copy, plus a three-process NYC taxi analysis pipeline.
- What you can do after
- Apply memory-sharing checks with NumPy and use Arrow Plasma to exchange objects between local processes.
Read how Fast Python: High performance techniques for large datasets explains shared memory →
Grokking Concurrency
A shared variable is a variable whose value can be accessed by multiple concurrent tasks.
- How it is used
- Uses shared memory for communication between concurrent tasks, including potentially large data exchanges.
- How deep it goes
- Explains access, benefits and synchronization risks with a threaded implementation and a design trade-off.
- Examples
- Illustrates a producer and consumer sharing an array, philosophers sharing a dumpling count, and tasks sharing matrices.
- What you can do after
- Evaluate shared memory against copying or message passing by considering communication efficiency and synchronization costs.
Also covered in
- Python Concurrency with asyncio
- Learn Concurrent Programming with Go
- C++ Concurrency in Action, Second Edition
- Transformers in Action
- AI Agents in Action, Second Edition
- Go in Action, Second Edition
- Advanced Algorithms and Data Structures
- AI GovernanceIn early access (MEAP)
What to know first
- threadA thread is a distinct sequence of execution with its own control flow and state, managed or scheduled by a processor, runtime, or operating system.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- compute-boundA computation is compute-bound when its performance is limited primarily by processing capacity rather than memory transfer, input/output, or other resources.
- CPUA central processing unit (CPU) is a computer processor that executes program instructions and performs general-purpose computational operations.
Related concepts
- memory hierarchypart of · A memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.
- message passingalternative to · Message passing is a communication technique in which independent entities exchange explicit messages to transmit information, coordinate computation, or update state.
- tilingused by · Tiling is an optimization technique that partitions data or computation into smaller blocks to improve locality, reuse, and memory efficiency.
- GPU memorycontrasts with · GPU memory is memory available to a graphics processing unit for storing data and state used in graphics or computation.
- channelalternative to · A channel is a medium or logical pathway for conveying information, messages, or other content between sources and recipients.
Questions about shared memory
What is shared memory?
Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.
Which Manning books explain shared memory?
Grokking GPU Programming, CUDA for LLMs, GPU Programming with Triton, Fast Python: High performance techniques for large datasets and Grokking Concurrency.
What should I know before learning shared memory?
Manning books that teach shared memory usually introduce thread, matrix-matrix multiplication, compute-bound and CPU first.
What is the difference between shared memory and message passing?
Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data. Message passing is a communication technique in which independent entities exchange explicit messages to transmit information, coordinate computation, or update state.
What is shared memory part of?
Shared memory is part of memory hierarchy. A memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




