liveBook · Manning

Effective memory bandwidth

As explained in Grokking GPU Programming

Effective bandwidth is the data-transfer rate a computation achieves after accounting for the efficiency of its memory-access pattern.

Effective memory bandwidth is the useful data delivered per second. When an access pattern fetches data that threads do not use, efficiency falls and so does effective bandwidth. This can make a kernel memory-bound even when it performs substantial arithmetic. section 10.3 section 10.12

A stride changes the bottleneck

Suppose a kernel performs 100 operations while reading and writing one float: 12.5 operations per byte. With stride-8 access at 12.5% efficiency, 900 GB/s peak bandwidth becomes about 112 GB/s effective bandwidth. The effective ridge point rises to about 89 operations per byte, so the kernel is memory-bound. Improving the access pattern can restore bandwidth. section 10.12

Figure 10.4 Strided access (stride 2): 32 threads, 2 transactions. The warp’s 32 threads (T0 … T31) each read every other memory element ([0], [2], …, [62]), so their addresses span 64 elements instead of 32. The two brackets below memory mark two 128-byte cache lines: T0 through T15 fall in the first, T16 through T31 in the second. Because the warp’s addresses straddle two cache lines, the hardware must issue two memory transactions instead of one, and only half of each transaction’s bytes are actually used. Following the Snapshot Rule, picture 32 arrows split across two cache lines, with every other slot empty. (In sectors: eight 32-byte sectors fetched for 128 useful bytes, the same 50%.)

See how stride-2 accesses make 32 threads use only half the bytes fetched in two memory transactions. section 10.6

How Grokking GPU Programming covers it

How deep it goes
Explains useful-byte efficiency with stride examples and its effect on the Roofline ridge point.
Examples
Illustrates stride-8 efficiency at 12.5% and stride-4 efficiency at 25%, with resulting bandwidth estimates.
What you can do after
Diagnose when wasted memory transactions lower effective bandwidth and raise a kernel’s effective ridge point.
How it connects
Connects effective bandwidth to arithmetic intensity, the Roofline model, and strided memory access.

Also explained in

Compare how 2 books explain effective memory bandwidth →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.