Memory traffic
Memory traffic is the amount of data transferred between memory and processing units during computation.
Grokking GPU Programming
Memory traffic is the amount of data transferred between memory and processing units during computation.
- How it is used
- Uses memory traffic to explain why naive GPU matrix multiplication waits on data and why tiling helps.
- How deep it goes
- Calculates per-thread loads and compares requested global traffic for naive and tiled kernels, including the aligned-dimension reduction rule.
- Examples
- Compares 4 GB of naive requested input traffic with 128 MB using 32 × 32 tiles for 1024 × 1024 and 1024 × 512 matrices.
- What you can do after
- Evaluate requested input traffic and arithmetic intensity for naive matrix multiplication, and estimate the reduction from tiling.
GPU Programming with Triton
Global memory traffic is the movement of data between GPU computation and global memory.
- How it is used
- Motivates kernel fusion as a way to reduce memory movement and keep the GPU computing rather than waiting on memory.
- How deep it goes
- Quantifies per-element memory accesses and float32 bytes for fused and unfused operations.
- Examples
- Counts transfers for an unfused SiLU-and-multiplication chain and its fused equivalent.
- What you can do after
- Evaluate memory costs by counting per-element loads and stores in fused and unfused kernels.
Read how GPU Programming with Triton explains memory traffic →
Also covered in
- Quantization and Fast InferenceIn early access (MEAP)
- CUDA for LLMs
What to know first
- tilingTiling is an optimization technique that partitions data or computation into smaller blocks to improve locality, reuse, and memory efficiency.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- GPU memoryGPU memory is memory available to a graphics processing unit for storing data and state used in graphics or computation.
- matrix-matrix multiplicationMatrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
Related concepts
- tilingaddressed by · Tiling is an optimization technique that partitions data or computation into smaller blocks to improve locality, reuse, and memory efficiency.
- bit precisionconfigured by · Bit precision is the number of bits used to represent a numerical value.
- kernel fusionaddressed by · Kernel fusion is an optimization that combines multiple computational operations into fewer execution units or passes, reducing overhead and often intermediate data movement.
- arithmetic intensitymeasured by · Arithmetic intensity is the ratio of computational operations performed to bytes of data transferred by a computation.
- naive matrix multiplication kernelproduced by · A naive matrix multiplication kernel computes each output element by accumulating the dot product of the corresponding row and column.
- quantizationaddressed by · Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
Questions about memory traffic
What is memory traffic?
Memory traffic is the amount of data transferred between memory and processing units during computation.
Which Manning books explain memory traffic?
Grokking GPU Programming and GPU Programming with Triton.
What should I know before learning memory traffic?
Manning books that teach memory traffic usually introduce tiling, attention, GPU memory and matrix-matrix multiplication first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

