liveBook · Manning

Chapter 10: Memory access patterns

What you will learn in chapter 10 of Grokking GPU Programming

This introduction identifies global-memory access patterns as a possible reason a correct, well-organized GPU kernel runs slowly.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction identifies global-memory access patterns as a possible reason a correct, well-organized GPU kernel runs slowly.

Key terms: global memory, memory access pattern, bottleneck

10.1 The memory bottleneck

This section explains how slow memory access can bottleneck GPU computation and introduces coalescing as a way to bundle thread requests.

Key terms: memory bottleneck, memory system, computation unit, memory coalescing, memory transaction, thread requests

10.2 Compute-bound or memory-bound?

This section uses a simple kernel to show how limited memory bandwidth can leave GPU computation waiting for data.

Key terms: FLOPS, TFLOPS, memory bandwidth, memory-bound, memory access patterns

10.3 Arithmetic intensity

This section uses arithmetic intensity and the Roofline model to estimate whether a kernel is limited by memory bandwidth or compute capacity.

Key terms: arithmetic intensity, FLOPs, bytes transferred, Roofline model, memory bandwidth, peak compute, ridge point, memory-bound, compute-bound, peak FLOPS

10.4 How GPUs really read memory

This section explains warp-level memory requests using a cache-line teaching model and contrasts it with modern sector-based transfers.

Key terms: warp, SIMT, cache line, memory transaction, coalesced access, scattered access, alignment, cache-line boundary, float, double

10.5 Coalesced access: the ideal case

This section explains coalesced memory access and introduces the Snapshot Rule for judging a warp’s accesses.

Key terms: coalesced access, warp, 128-byte cache line, transaction, Snapshot Rule, coalesced

10.6 Strided access: skipping elements

This section explains how strided memory access wastes bandwidth and raises the effective Roofline ridge point.

Key terms: strided access, stride, coalesced, memory transaction, cache line, bandwidth efficiency, aligned addresses, effective bandwidth, effective ridge point, Roofline model

10.7 Scattered access: the worst case

This section explains the cost of scattered GPU memory accesses and contrasts them with consecutive accesses in reverse order.

Key terms: scattered access, cache line, transactions, warp, coalesced, consecutive addresses, reverse, L2 caches, scattered accesses

10.8 Real-world access patterns

This section connects memory access patterns and performance to data layout and how threads traverse data.

Key terms: coalesced access, strided access, scattered access, data layout

10.9 AoS and SoA

This section compares AoS and SoA layouts and connects their access patterns to GPU coalescing, CPU locality, and PyTorch tensor strides.

Key terms: array of structures, AoS, structure of arrays, SoA, stride, coalescing, cache lines, per load instruction, cache reuse, memory-bound

10.10 Matrices

This section explains how matrix layout and thread assignment affect memory coalescing, and how to choose an efficient assignment for row sums.

Key terms: row-major order, column-major order, Snapshot Rule, sumRows, sumCols, coalesced, strided, warp, coalesced reads, parallel reduction

10.11 Common mistakes

The section identifies reliance on assumed kernel efficiency rather than measured data movement as a common source of two memory-related pitfalls.

Key terms: kernel efficiency, hardware moves bytes

10.12 Mistake 1: "My kernel does a lot of math, so it must be fast"

This section explains why arithmetic-heavy kernels can still be memory-bound and how to assess them using arithmetic intensity and effective bandwidth.

Key terms: arithmetic intensity, effective bandwidth, effective ridge point, memory-bound, strided, stride-8 layout, compute-bound

10.13 Mistake 2: Ignoring memory alignment

This section explains how misalignment and float3 layouts can reduce memory bandwidth, and how to address each case.

Key terms: coalesced, 128-byte boundary, cache lines, transaction, cudaMalloc, float3, float4, 16-byte vector loads, stride-3, SoA

10.14 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 9: The Stencil pattern Chapter 11: Shared memory →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.