liveBook · Manning

Memory coalescing

As explained in Grokking GPU Programming

Memory coalescing combines memory requests from GPU threads into fewer transactions when their accesses are suitably grouped.

A GPU kernel can have plenty of threads and still spend much of its time waiting for memory. Memory coalescing helps by combining suitably grouped thread requests into fewer memory transactions. Think of it as carpooling: several requests make one trip instead of lining up separately. section 10.1 section 10.0

To check an access, take a snapshot of the addresses requested at one instant by a warp: a group of 32 threads issuing a load together. The practical rule is to make consecutive threads access consecutive addresses. Looking at one thread’s path over time can hide what the whole warp is doing. section 10.5 section 10.4

Copying consecutive floats

In an array copy, thread 0 reads input[0], thread 1 reads input[1], and so on. If the warp’s 32 four-byte floats all fall in one 128-byte cache line, the simplified model counts one transaction. If the requests are scattered, they can require many more trips. section 10.5 section 10.4

Figure 10.3 Coalesced access: 32 threads, 1 transaction. The warp’s 32 threads (T0 … T31) each read the memory element whose index matches their thread number ([0] … [31]). The bracket below memory marks the span of one 128-byte cache line, which is exactly wide enough to hold all 32 four-byte floats. Because every thread’s address falls inside this single cache line, the hardware serves the entire warp with one memory transaction, the best possible outcome. This is the snapshot the Snapshot Rule asks you to picture: 32 arrows, all packed tightly into one cache line. (Counted in 32-byte sectors, the same read is four sectors covering one contiguous 128-byte span, still fully coalesced.)

See the warp’s 32 requests packed into one cache line. section 10.5

The 128-byte line is a teaching model. Modern GPUs also move data in 32-byte sectors: the aligned coalesced read above covers four sectors. Profiling tools such as Nsight Compute report sectors, so transaction counts depend on which unit is being counted. section 10.4

A matrix access that looks better from one thread

In a row-major matrix, elements are consecutive across each row. Assigning one thread per row lets each thread walk through consecutive elements, but at any instant neighboring warp threads read different rows, one matrix width apart. For a 1024-wide float matrix, consecutive threads’ first reads are 4,096 bytes apart. Assigning threads to consecutive columns instead makes their simultaneous reads consecutive, even though each individual thread walks down a column with a stride. To sum rows efficiently, assign threads along columns and use a parallel reduction within each row. section 10.10

Coalescing both sides of a transpose

A naive matrix transpose reads rows and writes columns, leaving one side uncoalesced. A tiled version loads a 32×32 tile into shared memory with coalesced reads, synchronizes, then writes it transposed with coalesced writes. Padding the shared tile to 32×33 avoids a 32-way bank conflict. The example assumes the matrix size is a multiple of 32; other sizes need bounds checks. section 11.9

__global__ void transposeTiled(const float* in, float* out, int n) {
    __shared__ float tile[32][33];    #A
    int x = blockIdx.x * 32 + threadIdx.x;
    int y = blockIdx.y * 32 + threadIdx.y;
    tile[threadIdx.y][threadIdx.x] = in[y * n + x];    #B
    __syncthreads();    #C
    x = blockIdx.y * 32 + threadIdx.x;
    y = blockIdx.x * 32 + threadIdx.y;
    out[y * n + x] = tile[threadIdx.x][threadIdx.y];    #D
}

Follow the load, synchronization barrier, and transposed write in the tiled kernel. section 11.9

How Grokking GPU Programming covers it

How deep it goes
Explains warp-level access snapshots and transaction counting, including the distinction between cache-line and sector models.
Examples
Uses an array copy, row- and column-wise matrix access, and a tiled matrix transpose.
What it assumes
Assumes familiarity with GPU kernels and the memory hierarchy while explaining warp-level access and the transaction model.
What you can do after
Apply warp-level address snapshots to identify uncoalesced accesses and choose thread assignments or shared-memory tiling.
How it connects
Connects coalescing to row-major layout, parallel reduction, shared memory, synchronization, and bank conflicts.

Also explained in

Compare how 3 books explain memory coalescing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.