Chapter 10: Memory access patterns
This introduction identifies global-memory access patterns as a possible reason a correct, well-organized GPU kernel runs slowly.
After this chapter you will be able to
- Memory Access Performance Bottleneck. Explain why a correct kernel with many well-organized threads can still be slow.
- Memory system bottleneck. Explain why the memory system can limit the pace of GPU computation.
- Memory coalescing as carpooling. Describe what memory coalescing does and why the section compares it to carpooling.
- Memory-bound increment example. Explain why the one-float increment example is memory-bound, using its bandwidth comparison and the implication for memory access.
- Arithmetic intensity calculation. Calculate arithmetic intensity as floating-point operations divided by all bytes read and written, and apply it to simple kernels.
- Roofline performance limits. Explain the two parts of the Roofline model and how attainable performance changes as arithmetic intensity increases.
- Ridge point classification. Calculate a GPU’s ridge point and compare kernel intensity with it to identify the likely memory or compute bottleneck.
- Warp access and coalescing. Explain how a warp issues memory instructions and why consecutive float accesses can require fewer transactions than scattered accesses.
- Alignment and data size. Describe how starting alignment and element size affect the number of transactions in the chapter's cache-line model.
- Coalesced access and transactions. Explain coalesced access using the section’s thread-to-element copy example and describe the resulting transaction in its ideal warp example.
- Warp access Snapshot Rule. Describe how to use the Snapshot Rule to judge whether an access is coalesced.
- Stride and access direction. Describe stride as the spacing between elements accessed by consecutive threads, and distinguish the strided reads from the coalesced writes in the example.
- Transaction and bandwidth waste. Trace the aligned stride-2 warp example and explain how larger strides reduce bandwidth efficiency.
- Effective Roofline ridge point. Explain how wasted transfers lower effective bandwidth and raise the effective ridge point, using the stride-4 example.
- Scattered access transaction cost. Explain why scattered accesses can require many transactions and waste transferred data compared with regular small-stride accesses.
- Reverse coalesced access. Apply the access-pattern distinction to explain why consecutive threads using n-1-idx access consecutive addresses in reverse rather than scattered or strided addresses.
- Layout and access patterns. Explain how data layout and the way threads walk through data affect memory access patterns and performance.
- Particle layout and coalescing. Explain how AoS and SoA arrange particle fields and how those layouts affect a warp reading the same field from consecutive particles. Apply the stride calculation to a three-float particle.
- Load efficiency and reuse. Trace the six-field AoS example's per-load memory efficiency and explain why its whole-kernel waste can be smaller.
- CPU and GPU locality. Explain why AoS can suit a CPU's particle-at-a-time access but be unfavorable for a GPU warp, and why the access pattern matters on both.
- PyTorch tensor strides. Describe what tensor.stride() reports, how transpose or stepped slicing can create strided access, and what tensor.contiguous() does.
- Matrix layout conventions. Distinguish row-major from column-major matrix layout and explain why the data producer’s convention matters in CUDA.
- Warp snapshot access patterns. Trace the simultaneous accesses of a warp in the row-major sumRows and sumCols examples, and distinguish them from a single thread’s traversal.
- Efficient row sum assignment. Describe how to arrange threads and combine their work for efficient row sums in a row-major matrix.
- Assumed Versus Measured Efficiency. Explain the common issue behind the two pitfalls concerning kernel math and data location.
- Arithmetic and memory bottlenecks. Explain why a kernel with many arithmetic operations can still run slowly, and describe how to assess its bottleneck.
- Warp Starting Address Alignment. Explain why a coalesced access can still lose bandwidth when its starting address is misaligned, and describe how to prevent it.
- Float3 Layout and Remedies. Explain the access problems in a float3 array and distinguish when padding to float4 or using separate component arrays helps.
What it builds on
- How the hardware runs your threads: warps and SMs (chapter 2) · Introduces warps and how hardware runs threads.
- The global index formula (chapter 3) · Threads in different blocks can be assigned unique global indices.
- Tree reduction: from concept to code (chapter 5) · Parallel reduction combines multiple values into one result.
- Where threads share data: shared memory (chapter 5) · Shared memory lets threads in a block share data.
Section by section
Introduction
This introduction identifies global-memory access patterns as a possible reason a correct, well-organized GPU kernel runs slowly.
- Memory Access Performance Bottleneck: Explain why a correct kernel with many well-organized threads can still be slow.
Why it matters: Launching many threads alone does not guarantee a speedup if their memory access holds the kernel back.
Key terms: global memory, memory access pattern, bottleneck
10.1 The memory bottleneck
This section explains how slow memory access can bottleneck GPU computation and introduces coalescing as a way to bundle thread requests.
- Memory system bottleneck: Explain why the memory system can limit the pace of GPU computation.
Why it matters: Fast computation cannot help much if threads spend their time waiting for data. - Memory coalescing as carpooling: Describe what memory coalescing does and why the section compares it to carpooling.
Why it matters: Bundling requests can reduce the pressure on the memory bottleneck.
Key terms: memory bottleneck, memory system, computation unit, memory coalescing, memory transaction, thread requests
10.2 Compute-bound or memory-bound?
This section uses a simple kernel to show how limited memory bandwidth can leave GPU computation waiting for data.
- Compute and bandwidth units: Distinguish FLOPS and TFLOPS from memory bandwidth using the example GPU’s figures.
Why it matters: These units let you compare a GPU’s computation capacity with its ability to supply data. - Memory-bound increment example: Explain why the one-float increment example is memory-bound, using its bandwidth comparison and the implication for memory access.
Why it matters: A high compute rate alone cannot make a kernel fast if memory cannot feed it.
Key terms: FLOPS, TFLOPS, memory bandwidth, memory-bound, memory access patterns
10.3 Arithmetic intensity
This section uses arithmetic intensity and the Roofline model to estimate whether a kernel is limited by memory bandwidth or compute capacity.
- Arithmetic intensity calculation: Calculate arithmetic intensity as floating-point operations divided by all bytes read and written, and apply it to simple kernels.
Why it matters: This calculation gives you a way to compare a kernel’s work with its memory traffic. - Roofline performance limits: Explain the two parts of the Roofline model and how attainable performance changes as arithmetic intensity increases.
Why it matters: The Roofline picture shows why more operations per byte can help only until the compute ceiling is reached. - Ridge point classification: Calculate a GPU’s ridge point and compare kernel intensity with it to identify the likely memory or compute bottleneck.
Why it matters: The ridge point turns arithmetic intensity into a practical first guess about where to look for a bottleneck. - Roofline diagnostic limitations: Describe why Roofline classification is a rule of thumb and how wasted bandwidth affects the effective ridge point.
Why it matters: A Roofline estimate is a useful starting point, but profiling can reveal the actual limiting factor.
Key terms: arithmetic intensity, FLOPs, bytes transferred, Roofline model, memory bandwidth, peak compute, ridge point, memory-bound, compute-bound, peak FLOPS
10.4 How GPUs really read memory
This section explains warp-level memory requests using a cache-line teaching model and contrasts it with modern sector-based transfers.
- Warp access and coalescing: Explain how a warp issues memory instructions and why consecutive float accesses can require fewer transactions than scattered accesses.
Why it matters: Accesses that fit together can serve a whole warp with fewer memory transactions. - Alignment and data size: Describe how starting alignment and element size affect the number of transactions in the chapter's cache-line model.
Why it matters: Consecutive accesses alone do not establish that a warp's data fits in one cache line. - Sector and teaching models: Distinguish the chapter's cache-line model from modern sector-based transfers and explain what the model is useful for.
Why it matters: The simplified model helps compare patterns without mistaking its counts for a direct speed prediction.
Key terms: warp, SIMT, cache line, memory transaction, coalesced access, scattered access, alignment, cache-line boundary, float, double
10.5 Coalesced access: the ideal case
This section explains coalesced memory access and introduces the Snapshot Rule for judging a warp’s accesses.
- Coalesced access and transactions: Explain coalesced access using the section’s thread-to-element copy example and describe the resulting transaction in its ideal warp example.
Why it matters: This ideal pattern shows how neighboring threads can make efficient use of a memory transaction. - Warp access Snapshot Rule: Describe how to use the Snapshot Rule to judge whether an access is coalesced.
Why it matters: Looking across the warp at one instant gives you the right view for judging its access pattern.
Key terms: coalesced access, warp, 128-byte cache line, transaction, Snapshot Rule, coalesced
10.6 Strided access: skipping elements
This section explains how strided memory access wastes bandwidth and raises the effective Roofline ridge point.
- Stride and access direction: Describe stride as the spacing between elements accessed by consecutive threads, and distinguish the strided reads from the coalesced writes in the example.
Why it matters: Identifying the stride of each access helps you spot where a kernel skips over data. - Transaction and bandwidth waste: Trace the aligned stride-2 warp example and explain how larger strides reduce bandwidth efficiency.
Why it matters: A small change in access spacing can make the GPU transfer data that no thread needs. - Effective Roofline ridge point: Explain how wasted transfers lower effective bandwidth and raise the effective ridge point, using the stride-4 example.
Why it matters: The access pattern can change whether a kernel is limited by memory or compute.
Key terms: strided access, stride, coalesced, memory transaction, cache line, bandwidth efficiency, aligned addresses, effective bandwidth, effective ridge point, Roofline model
10.7 Scattered access: the worst case
This section explains the cost of scattered GPU memory accesses and contrasts them with consecutive accesses in reverse order.
- Scattered access transaction cost: Explain why scattered accesses can require many transactions and waste transferred data compared with regular small-stride accesses.
Why it matters: This explains why scattered reads can consume much more memory bandwidth than the requested data suggests. - Reverse coalesced access: Apply the access-pattern distinction to explain why consecutive threads using n-1-idx access consecutive addresses in reverse rather than scattered or strided addresses.
Why it matters: This helps you classify a memory pattern by the relationship between thread addresses rather than by whether indices increase. - L2 cache mitigation: Describe how an L2 cache can partially absorb scattered accesses to small arrays.
Why it matters: This qualifies the worst-case transaction picture for small arrays in practice.
Key terms: scattered access, cache line, transactions, warp, coalesced, consecutive addresses, reverse, L2 caches, scattered accesses
10.8 Real-world access patterns
This section connects memory access patterns and performance to data layout and how threads traverse data.
- Layout and access patterns: Explain how data layout and the way threads walk through data affect memory access patterns and performance.
Why it matters: The same computation can perform differently depending on how its data is arranged and accessed.
Key terms: coalesced access, strided access, scattered access, data layout
10.9 AoS and SoA
This section compares AoS and SoA layouts and connects their access patterns to GPU coalescing, CPU locality, and PyTorch tensor strides.
- Particle layout and coalescing: Explain how AoS and SoA arrange particle fields and how those layouts affect a warp reading the same field from consecutive particles. Apply the stride calculation to a three-float particle.
Why it matters: The layout determines whether neighboring GPU threads read neighboring addresses. - Load efficiency and reuse: Trace the six-field AoS example's per-load memory efficiency and explain why its whole-kernel waste can be smaller.
Why it matters: A single-load efficiency figure does not necessarily describe the cost of an entire kernel. - CPU and GPU locality: Explain why AoS can suit a CPU's particle-at-a-time access but be unfavorable for a GPU warp, and why the access pattern matters on both.
Why it matters: A layout that works well for one access pattern may slow down another. - PyTorch tensor strides: Describe what tensor.stride() reports, how transpose or stepped slicing can create strided access, and what tensor.contiguous() does.
Why it matters: Tensor strides reveal whether a kernel's access along an axis is contiguous or strided. - Channels last distinction: Distinguish torch.channels_last from an AoS-to-SoA conversion.
Why it matters: Not every change in tensor layout is the AoS-to-SoA transformation.
Key terms: array of structures, AoS, structure of arrays, SoA, stride, coalescing, cache lines, per load instruction, cache reuse, memory-bound
10.10 Matrices
This section explains how matrix layout and thread assignment affect memory coalescing, and how to choose an efficient assignment for row sums.
- Matrix layout conventions: Distinguish row-major from column-major matrix layout and explain why the data producer’s convention matters in CUDA.
Why it matters: Knowing the actual layout lets you predict which matrix elements are adjacent in memory. - Warp snapshot access patterns: Trace the simultaneous accesses of a warp in the row-major sumRows and sumCols examples, and distinguish them from a single thread’s traversal.
Why it matters: A kernel’s memory efficiency depends on the warp’s simultaneous accesses, not just the path of one thread. - Efficient row sum assignment: Describe how to arrange threads and combine their work for efficient row sums in a row-major matrix.
Why it matters: Matching thread assignment to physical layout can make row summation more memory-efficient.
Key terms: row-major order, column-major order, Snapshot Rule, sumRows, sumCols, coalesced, strided, warp, coalesced reads, parallel reduction
10.11 Common mistakes
The section identifies reliance on assumed kernel efficiency rather than measured data movement as a common source of two memory-related pitfalls.
- Assumed Versus Measured Efficiency: Explain the common issue behind the two pitfalls concerning kernel math and data location.
Why it matters: Measuring data movement helps you avoid relying on assumptions about kernel efficiency.
Key terms: kernel efficiency, hardware moves bytes
10.12 Mistake 1: "My kernel does a lot of math, so it must be fast"
This section explains why arithmetic-heavy kernels can still be memory-bound and how to assess them using arithmetic intensity and effective bandwidth.
- Arithmetic and memory bottlenecks: Explain why a kernel with many arithmetic operations can still run slowly, and describe how to assess its bottleneck.
Why it matters: This helps you diagnose a slow kernel without assuming that more arithmetic guarantees speed. - Effective Ridge Point Example: Apply the worked example to explain how changing the access pattern changes the predicted bottleneck.
Why it matters: The example shows how the same calculation can have a different bottleneck when memory access becomes inefficient.
Key terms: arithmetic intensity, effective bandwidth, effective ridge point, memory-bound, strided, stride-8 layout, compute-bound
10.13 Mistake 2: Ignoring memory alignment
This section explains how misalignment and float3 layouts can reduce memory bandwidth, and how to address each case.
- Warp Starting Address Alignment: Explain why a coalesced access can still lose bandwidth when its starting address is misaligned, and describe how to prevent it.
Why it matters: Coalescing alone does not guarantee that a warp uses memory bandwidth efficiently. - Float3 Layout and Remedies: Explain the access problems in a float3 array and distinguish when padding to float4 or using separate component arrays helps.
Why it matters: The best layout depends on whether threads read whole elements or just one component.
Key terms: coalesced, 128-byte boundary, cache lines, transaction, cudaMalloc, float3, float4, 16-byte vector loads, stride-3, SoA
10.14 Summary
Where this is used later
- (chapter intro) (chapter 11) · The memory bottleneck can limit kernel speed.
- When your kernel is slow (chapter 11) · The impact of memory access on kernel performance.
- Introducing shared memory (chapter 11) · The memory bottleneck
- Understanding the problem (chapter 12) · The distinction between compute-bound and memory-bound work.
- Step 4: Synchronize again, then repeat (chapter 12) · Arithmetic intensity as operations per byte
- (chapter intro) (chapter 13) · GPU memory fetches and cache lines
- One value, 32 reads (chapter 13) · The cache-line model for GPU memory reads.
- The limitation of constant memory (chapter 13) · Scattered access patterns
Concepts explained in this chapter
- memory coalescingMemory coalescing combines memory requests from GPU threads into fewer transactions when their accesses are suitably grouped.
- strideMemory stride is the distance between memory locations accessed by successive elements of an access pattern.
- struct of arraysA structure of arrays stores each record field in a separate array, with corresponding positions belonging to the same record.
- snapshot ruleThe Snapshot Rule is a memory-access analysis technique that considers the locations accessed by all warp threads at the same moment.
- arithmetic intensityArithmetic intensity is the number of floating-point operations performed per byte transferred to and from memory.
- array of structsAn array of structures stores records as an array in which each element contains multiple fields belonging to one record.
- scattered memory access patternA scattered memory access pattern occurs when threads access memory locations that are not close together, often requiring separate memory transactions.
- ridge pointA ridge point is the arithmetic-intensity threshold that separates memory-bound from compute-bound performance for a given bandwidth and compute rate.
- effective memory bandwidthEffective bandwidth is the data-transfer rate a computation achieves after accounting for the efficiency of its memory-access pattern.
- memory transactionA memory transaction is a unit of memory service that processes one or more requests between a processor and memory.
- floating-point operation countA floating-point operation is an arithmetic operation performed on floating-point values.
- memory boundA memory-bound kernel is a GPU kernel whose performance is limited primarily by memory data transfer.
- matrixA matrix is a two-dimensional arrangement of elements organized by rows and columns.
- row-major orderRow-major order is a multidimensional array layout in which elements of each row are contiguous in memory.
- compute-boundCompute-bound describes a computation whose performance is limited primarily by the processor's computational capacity.
- alignmentMemory alignment is the placement of data at an address satisfying the boundary requirements of its type or hardware access.
- memory bandwidthMemory bandwidth is the rate at which data can be transferred between a processor and memory.
- floating-point operations per secondFLOPS measures the rate of floating-point operations performed per second.
- column-major orderColumn-major order is a multidimensional array layout in which elements of each column are contiguous in memory.
- reduction operationThe reduction pattern combines many input values into one result using an associative operation, often by combining partial results in parallel.
- L2 cacheAn L2 cache is a hardware-managed cache that stores copies of data to reduce the cost of memory access.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
