liveBook · Manning

Memory coalescing

Also called coalesced access, coalesced memory access

Memory coalescing is a GPU access pattern in which nearby threads access adjacent memory locations so their requests can be combined into fewer memory transactions.

How 3 Manning books explain it

CUDA for LLMs

Memory coalescing is a GPU memory-access technique in which neighboring threads access neighboring memory locations so requests can be combined.

How it is used
Uses coalescing to fix bandwidth-wasting access patterns in transformer kernels before further optimization.
How deep it goes
Traces warp addresses and thread mappings through cache-line behavior, kernel code, and measured performance, then extends the approach to tiled loads.
Examples
Works through row-wise GEMV, thread-remapped GEMM, and fused softmax.
What you can do after
Apply it via CUDA thread remapping and evaluate the result with realistic-size profiling.

Read how CUDA for LLMs explains memory coalescing →

GPU Programming with Triton

Memory coalescing is an access pattern in which neighboring parallel threads access adjacent memory locations so those accesses can be combined efficiently.

How it is used
Analyzes how warp-level global-memory access patterns affect transaction counts and GPU performance.
How deep it goes
Quantifies 32-lane FP32 accesses in 32-byte sectors, comparing contiguous and strided patterns.
Examples
Compares contiguous and stride-32 FP32 loads, then contrasts contiguous and strided transpose stores.
What you can do after
Evaluate warp access efficiency by comparing requested addresses with the memory sectors they occupy.

Read how GPU Programming with Triton explains memory coalescing →

Grokking GPU Programming

Memory coalescing combines memory requests from GPU threads into fewer transactions when their accesses are suitably grouped.

How it is used
Reduces global-memory bottlenecks by grouping simultaneous GPU thread requests into fewer transactions.
How deep it goes
Explains warp-level access snapshots and transaction counting, including the distinction between cache-line and sector models.
Examples
Uses an array copy, row- and column-wise matrix access, and a tiled matrix transpose.
What you can do after
Apply warp-level address snapshots to identify uncoalesced accesses and choose thread assignments or shared-memory tiling.

Read how Grokking GPU Programming explains memory coalescing →

Also covered in

What to know first

Related concepts

Questions about memory coalescing

What is memory coalescing?

Memory coalescing is a GPU access pattern in which nearby threads access adjacent memory locations so their requests can be combined into fewer memory transactions.

Which Manning books explain memory coalescing?

CUDA for LLMs, GPU Programming with Triton and Grokking GPU Programming.

What should I know before learning memory coalescing?

Manning books that teach memory coalescing usually introduce GPU memory, memory transaction, shared memory and warp first.

What is the difference between memory coalescing and scattered memory access pattern?

Memory coalescing is a GPU access pattern in which nearby threads access adjacent memory locations so their requests can be combined into fewer memory transactions. A scattered memory access pattern occurs when threads access memory locations that are not close together, often requiring separate memory transactions.

What is memory coalescing also called?

Memory coalescing is also called coalesced access or coalesced memory access.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.