liveBook · Manning

What you will learn in Grokking GPU Programming

With Examples in CUDA

by SungHwan Yun · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Why parallel programming?

This introduction explains why automatic hardware speedups gave way to multicore processors and identifies GPUs as a large source of processing units.

1.1 The end of the “Free Lunch” · 1.2 The parallel computing revolution · 1.3 A family of parallel approaches · 1.4 Sequential vs. parallel thinking · 1.5 Caution: not everything can be parallelized · 1.6 Why GPUs became essential · 1.7 Why learn CUDA? · 1.8 Your lab: swforces.com · 1.9 A preview of the patterns ahead · 1.10 Summary

What you learn in chapter 1 →

Chapter 2: Your first parallel program

This introduction previews the CUDA concepts covered in the chapter and the grading-room analogy that will guide the first program.

2.1 Hello, Parallel World! · 2.2 The headers: plugging in the toolkit · 2.3 The kernel function: a function that runs in parallel · 2.4 The launch syntax: starting those threads · 2.5 Synchronization: you have to wait · 2.6 Each thread has its own job · 2.7 Meet threadIdx · 2.8 How the hardware runs your threads: warps and SMs · 2.9 Squaring an array · 2.10 The function signature · 2.11 Thread index · 2.12 Bounds check · 2.13 The actual work · 2.14 Moving data: CPU to GPU and back · 2.15 Step 1: Allocate device memory (set up desks in the grading room) · 2.16 Step 2: Copy host to device (carry the exams to the students) · 2.17 Step 3: Launch kernel (let the students grade) · 2.18 Step 4: Copy device to host (collect the graded exams) · 2.19 Step 5: Free memory (clear the desks out of the room) · 2.20 Common mistakes · 2.21 Mistake 1: Forgetting to synchronize · 2.22 Mistake 2: Copying data the wrong way · 2.23 Mistake 3: Missing bounds check · 2.24 Summary

What you learn in chapter 2 →

Chapter 3: Scaling up: from one block to many

This introduction explains scaling array processing beyond one block by spreading threads across multiple blocks.

3.1 What’s wrong with one block? · 3.2 The three-level hierarchy: grids, blocks, and threads · 3.3 Why the thread index alone isn’t enough · 3.4 The global index formula · 3.5 The global index as a student ID · 3.6 Tracing through the numbers · 3.7 Your first multi-block kernel · 3.8 Launching the right number of threads · 3.9 Part A: How many blocks do I need? · 3.10 Part B: How big should each block be? · 3.11 The multi-block kernel recipe · 3.12 Common mistakes · 3.13 Mistake 1: Using threadIdx directly · 3.14 Mistake 2: Truncating instead of rounding up · 3.15 Mistake 3: Swapping the grid and block sizes · 3.16 Summary

What you learn in chapter 3 →

Chapter 4: The Map pattern

This introduction defines the Map pattern, illustrates its independent per-element work, and relates it to GPU execution.

4.1 What makes a problem “mappable”? · 4.2 Where you’ve seen Map before · 4.3 One thread, one element · 4.4 Map in action: vector addition · 4.5 Thinking in parallel · 4.6 Breaking it down, line by line · 4.7 Map in action: scalar vector multiplication · 4.8 Thinking in parallel · 4.9 Breaking it down, line by line · 4.10 Map in action: vector ReLU · 4.11 Thinking in parallel · 4.12 Breaking it down, line by line · 4.13 Common mistakes · 4.14 Mistake 1: Accessing another thread’s element · 4.15 Mistake 2: Accidentally overwriting your input · 4.16 Summary

What you learn in chapter 4 →

Chapter 5: The Reduction pattern

This introduction contrasts reduction with Map and illustrates parallel summation through partial sums and pairing rounds.

5.1 A million values, one answer · 5.2 You already know reduction · 5.3 Use cases for reduction · 5.4 Why naive parallelism fails · 5.5 The sequential approach · 5.6 The shared-variable trap: many threads, one counter · 5.7 Tree reduction: from concept to code · 5.8 The tournament analogy · 5.9 Where threads share data: shared memory · 5.10 Building the kernel step by step · 5.11 Act 1: Everyone writes on the whiteboard · 5.12 Act 2: The tournament begins · 5.13 Act 3: Report the block’s result · 5.14 The barrier: __syncthreads() · 5.15 The last piece: atomic operations · 5.16 Putting it all together: the complete kernel · 5.17 Beyond addition: max, min, and more · 5.18 Reduction in NumPy and PyTorch · 5.20 PyTorch · 5.21 Common mistakes · 5.22 Mistake 1: Missing __syncthreads() · 5.23 Mistake 2: Forgetting to initialize result · 5.24 Mistake 3: Regular add instead of atomic · 5.25 Mistake 4: Shared memory size mismatch · 5.26 Summary

What you learn in chapter 5 →

Chapter 6: The Scan pattern: Hillis-Steele

Introduces scan as a way to compute running results and contrasts its output and algorithmic tradeoffs with reduction and work-efficient scanning.

6.1 Understanding the Scan pattern · 6.2 The Hillis-Steele algorithm · 6.3 Round 0: starting state · 6.4 Round 1: stride = 1 · 6.5 Round 2: stride = 2 · 6.6 Round 3: stride = 4 · 6.7 Why does this work? · 6.8 Programming the Hillis-Steele algorithm with CUDA · 6.9 Part 1: Setup with shared memory and the double buffer · 6.10 Part 2: Load input into shared memory · 6.11 Part 3: The main loop, doubling stride · 6.12 Part 4: The parallel add with boundary check · 6.13 Part 5: Write the result to global memory · 6.14 Complete Hillis-Steele inclusive scan · 6.15 Common mistakes · 6.16 Mistake 1: Reading and writing the same array · 6.17 Mistake 2: Missing boundary check · 6.18 Mistake 3: Forgetting __syncthreads() · 6.19 Summary

What you learn in chapter 6 →

Chapter 7: Work-efficient scan: Blelloch

This introduction contrasts Blelloch’s work-efficient scan with Hillis-Steele and recaps the scan concepts and Hillis-Steele mechanism.

7.1 The Blelloch algorithm · 7.2 Up-sweep: building the reduction tree · 7.3 Seed and down-sweep: distributing the prefix sums · 7.4 Why does this work? · 7.5 Programming the Blelloch algorithm with CUDA · 7.6 Part 1: Load data into shared memory · 7.7 Part 2: Up-sweep (the reduction phase) · 7.8 Part 3: Clear the root (start of the down-sweep) · 7.9 Part 4: Down-sweep (the distribution phase) · 7.10 Part 5: Write the result back · 7.11 Complete Blelloch exclusive scan · 7.12 Hillis-Steele versus Blelloch · 7.13 Scan in the real world · 7.14 Stream compaction · 7.15 Memory allocation · 7.16 Radix sort · 7.17 Partitioning · 7.18 Scan in PyTorch · 7.19 Common mistakes · 7.20 Mistake 1: Mixing devices in PyTorch scans · 7.21 Summary

What you learn in chapter 7 →

Chapter 8: The Histogram pattern

This introduction illustrates lost updates in shared results and contrasts them with earlier patterns that reduce write collisions.

8.1 The conflict problem · 8.2 One counter, a thousand threads · 8.3 Making read-modify-write indivisible · 8.4 The CUDA atomic toolkit · 8.5 From one counter to 256 bins · 8.6 Why parallel histograms are hard · 8.7 A first fix: global atomics · 8.8 A faster fix: privatization in shared memory · 8.9 Common mistakes · 8.10 Mistake 1: Forgetting atomics · 8.11 Mistake 2: Conditional __syncthreads() (the deadlock trap) · 8.12 Summary

What you learn in chapter 8 →

Chapter 9: The Stencil pattern

Introduces stencil computation through a parallel image blur and contrasts its neighbor reads with histogram write conflicts.

9.1 What a stencil is: a window around every element · 9.2 A 1D stencil in code · 9.3 Stepping up to 2D · 9.4 From averages to weights · 9.5 How fast is the stencil? · 9.6 Handling the boundaries · 9.7 Clamp · 9.8 Zero padding · 9.9 Skip · 9.10 Wrap-around · 9.11 Where you’ll use the Histogram and Stencil patterns · 9.12 Common mistakes · 9.13 Mistake 1: Missing boundary checks · 9.14 Mistake 2: In-place stencil (the sneaky one) · 9.15 Summary

What you learn in chapter 9 →

Chapter 10: Memory access patterns

This introduction identifies global-memory access patterns as a possible reason a correct, well-organized GPU kernel runs slowly.

10.1 The memory bottleneck · 10.2 Compute-bound or memory-bound? · 10.3 Arithmetic intensity · 10.4 How GPUs really read memory · 10.5 Coalesced access: the ideal case · 10.6 Strided access: skipping elements · 10.7 Scattered access: the worst case · 10.8 Real-world access patterns · 10.9 AoS and SoA · 10.10 Matrices · 10.11 Common mistakes · 10.12 Mistake 1: "My kernel does a lot of math, so it must be fast" · 10.13 Mistake 2: Ignoring memory alignment · 10.14 Summary

What you learn in chapter 10 →

Chapter 11: Shared memory

This introduction explains why shared memory can speed up GPU computation and identifies the common collaboration pattern used in earlier kernels.

11.1 When your kernel is slow · 11.2 Introducing shared memory · 11.3 The 3-Step Recipe · 11.4 Producers and consumers · 11.5 Applying the 3-Step Recipe · 11.6 Bank conflicts · 11.7 Predicting bank conflicts from the access pattern · 11.8 Avoiding bank conflicts with padding · 11.9 A first taste of tiling: transpose · 11.10 Common mistakes · 11.11 Mistake 1: Bank conflicts in column access · 11.12 Mistake 2: Exceeding shared memory limits · 11.13 Mistake 3: Reading uninitialized shared memory · 11.14 Summary

What you learn in chapter 11 →

Chapter 12: Parallel matrix multiplication

This introduction explains the data bottleneck in naive matrix multiplication and how tiling with shared memory can reduce repeated global-memory access.

12.1 Understanding the problem · 12.2 The solution: tiled matrix multiplication · 12.3 Tiles: breaking a matrix into bite-sized chunks · 12.4 Walkthrough of a tiled kernel · 12.5 Step 1: Load tiles into shared memory · 12.6 Step 2: Synchronize · 12.7 Step 3: Compute using shared memory · 12.8 Step 4: Synchronize again, then repeat · 12.9 In production: use cuBLAS · 12.10 Common mistakes · 12.11 Mistake 1: Missing the second __syncthreads() · 12.12 Mistake 2: Row-major vs column-major confusion · 12.13 Mistake 3: Tile size over the limits · 12.14 Summary

What you learn in chapter 12 →

Chapter 13: Read-only data optimization

This introduction explains why repeated warp-wide reads can strain ordinary caches and how constant memory provides a dedicated cache for read-only data.

13.1 One value, 32 reads · 13.2 How constant memory works · 13.3 Constant memory within the hierarchy · 13.4 The three steps for using constant memory · 13.5 The broadcast mechanism · 13.6 How broadcast works, step by step · 13.7 The limitation of constant memory · 13.8 Putting it all together: constant memory array scaling · 13.9 Step 1: Declare in constant memory · 13.10 Step 2: Copy data to constant memory from the host · 13.11 Step 3: Use it in the kernel · 13.12 Common mistakes · 13.13 Mistake 1: Exceeding 64 KB · 13.14 Mistake 2: Using the wrong copy function · 13.15 Mistake 3: Using const instead of __constant__ · 13.16 Summary

What you learn in chapter 13 →

Chapter 14: Warps and occupancy

Introduces warp-based thread execution and its performance implications when threads take different paths.

14.1 The marching band inside your GPU · 14.2 When the band splits: branch divergence · 14.3 Three strategies for managing divergence · 14.4 Which strategy should you choose? · 14.5 Filling the stage: occupancy · 14.6 What limits occupancy: three resources · 14.7 Occupancy is a means, not a goal · 14.8 Worked example: checking all three constraints · 14.9 Common mistakes · 14.10 Mistake 1: Assuming cross-warp execution order · 14.11 Mistake 2: Blindly maximizing occupancy · 14.12 Summary

What you learn in chapter 14 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.