liveBook · Manning

Memory traffic

Memory traffic is the amount of data transferred between memory and processing units during computation.

How 2 Manning books explain it

Grokking GPU Programming

Memory traffic is the amount of data transferred between memory and processing units during computation.

How it is used
Uses memory traffic to explain why naive GPU matrix multiplication waits on data and why tiling helps.
How deep it goes
Calculates per-thread loads and compares requested global traffic for naive and tiled kernels, including the aligned-dimension reduction rule.
Examples
Compares 4 GB of naive requested input traffic with 128 MB using 32 × 32 tiles for 1024 × 1024 and 1024 × 512 matrices.
What you can do after
Evaluate requested input traffic and arithmetic intensity for naive matrix multiplication, and estimate the reduction from tiling.

Read how Grokking GPU Programming explains memory traffic →

GPU Programming with Triton

Global memory traffic is the movement of data between GPU computation and global memory.

How it is used
Motivates kernel fusion as a way to reduce memory movement and keep the GPU computing rather than waiting on memory.
How deep it goes
Quantifies per-element memory accesses and float32 bytes for fused and unfused operations.
Examples
Counts transfers for an unfused SiLU-and-multiplication chain and its fused equivalent.
What you can do after
Evaluate memory costs by counting per-element loads and stores in fused and unfused kernels.

Read how GPU Programming with Triton explains memory traffic →

Also covered in

What to know first

Related concepts

Questions about memory traffic

What is memory traffic?

Memory traffic is the amount of data transferred between memory and processing units during computation.

Which Manning books explain memory traffic?

Grokking GPU Programming and GPU Programming with Triton.

What should I know before learning memory traffic?

Manning books that teach memory traffic usually introduce tiling, attention, GPU memory and matrix-matrix multiplication first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.