liveBook · Manning

Shared memory

Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.

How 5 Manning books explain it

Grokking GPU Programming

Shared memory is a fast, explicitly managed memory space accessible to threads within a GPU thread block.

How it is used
Uses shared memory for block-level data reuse, cooperative loading, and thread communication in parallel GPU algorithms.
How deep it goes
Explains declarations, synchronization hazards, capacity limits, bank conflicts, and tiled kernels, with worked implementations and common mistakes.
Examples
Traces a padded reduction and compares global-memory traffic for naive and tiled matrix multiplication.
What you can do after
Apply it via CUDA to stage data, coordinate block threads, and diagnose initialization, synchronization, and capacity mistakes.

Read how Grokking GPU Programming explains shared memory →

CUDA for LLMs

Shared memory is a low-latency on-chip GPU memory region used for cooperation among threads.

How it is used
Uses block-local shared memory to reduce global-memory traffic and coordinate threads in deep-learning kernels.
How deep it goes
Explains allocation, cooperative loading, synchronization, reductions, tiling, and implicit tensor-core transfers in CUDA examples.
Examples
Works through GEMV reduction, tiled GEMM, fused LayerNorm, and Flash Attention tiles.
What you can do after
Build shared-memory CUDA kernels that cooperate on reductions and reuse data across tiled or fused computations.

Read how CUDA for LLMs explains shared memory →

GPU Programming with Triton

Shared memory is explicitly managed on-chip GPU memory visible to all threads in a single block.

How it is used
Motivates fast on-chip storage for thread cooperation and data reuse when global-memory access limits GPU performance.
How deep it goes
Explains staging, synchronization, access-pattern changes, and cache partitioning, while leaving Triton storage decisions to compiler lowering.
Examples
Uses matrix multiplication tiles and inter-warp partial-result exchange in reductions and scans.
What you can do after
Evaluate/understand when shared storage can reduce global-memory reads and how its capacity competes with L1 cache.

Read how GPU Programming with Triton explains shared memory →

Fast Python: High performance techniques for large datasets

How it is used
Uses managed shared memory for local process pipelines and array sharing to avoid unnecessary data copies.
How deep it goes
Demonstrates array sharing checks and a Plasma pipeline while deferring low-level shared-memory implementation.
Examples
Uses a flipped image view and copy, plus a three-process NYC taxi analysis pipeline.
What you can do after
Apply memory-sharing checks with NumPy and use Arrow Plasma to exchange objects between local processes.

Read how Fast Python: High performance techniques for large datasets explains shared memory →

Grokking Concurrency

A shared variable is a variable whose value can be accessed by multiple concurrent tasks.

How it is used
Uses shared memory for communication between concurrent tasks, including potentially large data exchanges.
How deep it goes
Explains access, benefits and synchronization risks with a threaded implementation and a design trade-off.
Examples
Illustrates a producer and consumer sharing an array, philosophers sharing a dumpling count, and tasks sharing matrices.
What you can do after
Evaluate shared memory against copying or message passing by considering communication efficiency and synchronization costs.

Read how Grokking Concurrency explains shared memory →

Also covered in

What to know first

Related concepts

Questions about shared memory

What is shared memory?

Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data.

Which Manning books explain shared memory?

Grokking GPU Programming, CUDA for LLMs, GPU Programming with Triton, Fast Python: High performance techniques for large datasets and Grokking Concurrency.

What should I know before learning shared memory?

Manning books that teach shared memory usually introduce thread, matrix-matrix multiplication, compute-bound and CPU first.

What is the difference between shared memory and message passing?

Shared memory is a memory region accessible to multiple processes, threads, or other concurrent execution contexts for sharing data. Message passing is a communication technique in which independent entities exchange explicit messages to transmit information, coordinate computation, or update state.

What is shared memory part of?

Shared memory is part of memory hierarchy. A memory hierarchy is an arrangement of storage levels with differing capacities, access speeds, latencies, costs, or persistence.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.