Chapter 3: Scaling up: from one block to many
This introduction explains scaling array processing beyond one block by spreading threads across multiple blocks.
After this chapter you will be able to
- Scaling beyond one block. Explain why spreading threads across multiple blocks lets the same kernel handle arrays larger than a single block can cover, using the classroom analogy.
- Single block scaling limit. Explain the per-block thread limit and how multiple blocks allow a larger launch.
- Rejected launch diagnosis. Describe how to detect a rejected launch and interpret an invalid configuration error.
- Grid Block Thread Hierarchy. Describe the three levels of a CUDA launch and interpret the block and thread counts in its launch configuration.
- Block Scheduling and Portability. Explain how blocks are assigned to SMs and why the same block-based kernel can run on GPUs with different numbers of SMs.
- Independent Block Execution Order. Explain why a kernel must not rely on one block finishing before another.
- Cross Block Synchronization and Reduction. Explain the cross-block synchronization limitation and describe the two-step idea for combining an array’s values.
- Block Local Index Overlap. Explain why using threadIdx.x alone as an array index fails in a multi-block launch, using the three-block example.
- Unique Global Index Inputs. Describe what blockIdx.x and blockDim.x contribute to identifying a thread's global array position, and trace the section's block 2, thread 1 example.
- Global thread index formula. Explain how to combine the built-in x components to give each thread in a grid a unique global index.
- Global thread index analogy. Explain how classroom and desk numbers correspond to the CUDA global thread index, including why numbering starts at zero.
- Global thread index tracing. Apply the global index formula to threads in a multi-block launch and explain how their indices cover the array.
- GPU memory tier tradeoffs. Distinguish registers, shared memory, and global memory by visibility and the size-versus-speed tradeoff.
- Guarded array doubling. Explain how the multi-block kernel doubles array elements safely using each thread’s unique index and a bounds check.
- Block coverage and excess. Determine the blocks, total threads, and excess threads for an array when each block has a fixed number of threads, and explain why rounding up is necessary.
- Ceiling division for blocks. Explain and apply ceiling division to calculate the number of blocks needed for a one-dimensional input.
- Empty input launch handling. Describe what to do when the input has zero elements and why.
- Block size and warp utilization. Explain why 256 threads per block is a useful starting choice and when to reconsider it.
- Multi-block array kernel recipe. Apply the four-step recipe to launch a kernel that processes a one-dimensional array.
- Repeated local index bug. Explain the faulty indexing behind a kernel that repeatedly modifies the first block’s elements while leaving later elements unchanged.
- Thread index overlap. Explain why using threadIdx.x directly produces incorrect results with multiple blocks, how to recognize the problem, and how a global index fixes it.
- Rounding Up the Block Count. Explain why integer division can launch too few blocks when the array size is not a multiple of the block size, and describe how ceiling division fixes it.
- Grid and block argument order. Explain the correct order of CUDA launch arguments and the consequences of swapping them.
What it builds on
- Squaring an array (chapter 2) · Processing an array by assigning an element to each thread.
- The launch syntax: starting those threads (chapter 2) · The launch syntax specifies the number of blocks and threads per block.
- How the hardware runs your threads: warps and SMs (chapter 2) · Introduces streaming multiprocessors as the hardware that runs threads.
- Synchronization: you have to wait (chapter 2) · Synchronization can be used to wait for kernel work to finish.
- Meet threadIdx (chapter 2) · The role of threadIdx in identifying a thread.
- Step 1: Allocate device memory (set up desks in the grading room) (chapter 2) · Device memory allocation with cudaMalloc.
- Bounds check (chapter 2) · The bounds check protects array accesses.
- Step 4: Copy device to host (collect the graded exams) (chapter 2) · Device-to-host copying brings results back to the host.
Section by section
Introduction
This introduction explains scaling array processing beyond one block by spreading threads across multiple blocks.
- Scaling beyond one block: Explain why spreading threads across multiple blocks lets the same kernel handle arrays larger than a single block can cover, using the classroom analogy.
Why it matters: This is the basic shift that lets your GPU programs handle larger problems.
Key terms: block, thread, scaling up, kernel
3.1 What’s wrong with one block?
This section explains why a single CUDA block cannot scale to large workloads, how multiple blocks address the limit, and how to detect a rejected launch.
- Single block scaling limit: Explain the per-block thread limit and how multiple blocks allow a larger launch.
Why it matters: This limit explains why larger workloads need more than one block. - Block coordination hardware rationale: Explain why a block must fit on one streaming multiprocessor and how independent blocks avoid that same constraint.
Why it matters: The hardware rationale makes the per-block limit less arbitrary. - Rejected launch diagnosis: Describe how to detect a rejected launch and interpret an invalid configuration error.
Why it matters: Checking launch errors helps you find a kernel that never actually ran.
Key terms: block, threads, launch, streaming multiprocessor, SM, shared memory, synchronize, cudaGetLastError(), cudaErrorInvalidConfiguration, launch dimensions
3.2 The three-level hierarchy: grids, blocks, and threads
This section explains CUDA’s grid-block-thread hierarchy, block scheduling, and the implications for ordering and combining work across blocks.
- Grid Block Thread Hierarchy: Describe the three levels of a CUDA launch and interpret the block and thread counts in its launch configuration.
Why it matters: This hierarchy lets you describe how a kernel launch divides work among threads. - Block Scheduling and Portability: Explain how blocks are assigned to SMs and why the same block-based kernel can run on GPUs with different numbers of SMs.
Why it matters: Block scheduling allows a kernel’s work to scale across different GPUs without changing its division into blocks. - Independent Block Execution Order: Explain why a kernel must not rely on one block finishing before another.
Why it matters: Depending on a block execution order that CUDA does not guarantee can make a kernel incorrect. - Cross Block Synchronization and Reduction: Explain the cross-block synchronization limitation and describe the two-step idea for combining an array’s values.
Why it matters: Recognizing this limitation helps you understand why combining results across blocks needs a different strategy from independent per-element work.
Key terms: grid, block, thread, numBlocks, threadsPerBlock, SM, portable, blocks, independent work, barrier
3.3 Why the thread index alone isn’t enough
This section explains why block-local thread indices overlap and introduces the information needed to give threads distinct array positions.
- Block Local Index Overlap: Explain why using threadIdx.x alone as an array index fails in a multi-block launch, using the three-block example.
Why it matters: Recognizing repeated local indices helps you avoid assigning multiple blocks the same work. - Unique Global Index Inputs: Describe what blockIdx.x and blockDim.x contribute to identifying a thread's global array position, and trace the section's block 2, thread 1 example.
Why it matters: These inputs let you distinguish threads across blocks and assign each one its own element. - Concurrent Write Race: Explain what can happen when threads in different blocks try to add to the same element, and identify the kind of operation CUDA provides for safe combination.
Why it matters: Overlapping writes can produce lost updates rather than the combined result you expect.
Key terms: threadIdx.x, index overlap problem, global index, blockIdx.x, blockDim.x, race condition, atomic operations
3.4 The global index formula
This section introduces the one-dimensional global thread index and the chapter’s one-thread-per-element model.
- Global thread index formula: Explain how to combine the built-in x components to give each thread in a grid a unique global index.
Why it matters: A global index lets threads across different blocks be distinguished. - One thread per element: Describe the thread-to-data model used in this chapter and how it relates to other possible mappings.
Why it matters: This model makes the global index useful for assigning array work while leaving room for other mappings.
Key terms: global index, blockIdx.x, blockDim.x, threadIdx.x, one thread per element, map threads to data
3.5 The global index as a student ID
This section explains global thread indexing through a student ID analogy and connects the same grouping formula to flattening a two-dimensional array.
- Global thread index analogy: Explain how classroom and desk numbers correspond to the CUDA global thread index, including why numbering starts at zero.
Why it matters: This analogy makes it easier to assign each thread its own position across multiple blocks. - Row-major flattening pattern: Describe how the group-size indexing pattern converts a row and column into a position in a flat array.
Why it matters: The same indexing idea applies when rows are laid out one after another in a flat array.
Key terms: global index, blockIdx.x, blockDim.x, threadIdx.x, row-major flattening, flat array, row, width, column
3.6 Tracing through the numbers
The section traces global thread indices across blocks and introduces the GPU memory hierarchy.
- Global thread index tracing: Apply the global index formula to threads in a multi-block launch and explain how their indices cover the array.
Why it matters: Tracing indices shows how multiple blocks can assign one thread to each array element without gaps or overlaps. - GPU memory tier tradeoffs: Distinguish registers, shared memory, and global memory by visibility and the size-versus-speed tradeoff.
Why it matters: The memory tiers give threads different places to store data, with different visibility, capacity, and speed.
Key terms: global index, blockIdx.x, blockDim.x, threadIdx.x, registers, shared memory, global memory, memory hierarchy
3.7 Your first multi-block kernel
This section applies global indexing and a bounds check to a multi-block array-doubling kernel, showing how to cover an array safely when blocks do not divide its size evenly.
- Guarded array doubling: Explain how the multi-block kernel doubles array elements safely using each thread’s unique index and a bounds check.
Why it matters: The guard lets extra threads run without accessing memory beyond the array. - Block coverage and excess: Determine the blocks, total threads, and excess threads for an array when each block has a fixed number of threads, and explain why rounding up is necessary.
Why it matters: Choosing enough blocks covers every element while identifying the extra threads that must be guarded. - Boundary warp divergence: Describe the small inefficiency that can arise in a warp crossing the array boundary and contrast it with the risk of omitting the bounds check.
Why it matters: A small boundary inefficiency is preferable to unsafe memory access.
Key terms: global index, bounds check, if (idx < n), out-of-bounds access, block size, blocks, threads, idle threads, warp, branch divergence
3.8 Launching the right number of threads
Introduces the upcoming calculation of block count and block size for a kernel launch.
3.9 Part A: How many blocks do I need?
This section explains how to calculate the number of CUDA blocks using ceiling division and handle empty inputs safely.
- Ceiling division for blocks: Explain and apply ceiling division to calculate the number of blocks needed for a one-dimensional input.
Why it matters: This calculation gives the input enough threads without adding unnecessary whole blocks. - Empty input launch handling: Describe what to do when the input has zero elements and why.
Why it matters: Skipping the launch avoids an invalid CUDA launch when there is no work. - Alternative formula edge case: Distinguish the alternative block-count formula from the recommended formula when n is zero.
Why it matters: An apparently equivalent formula can produce a wrong block count for empty input.
Key terms: ceiling division, numBlocks, threadsPerBlock, integer division, n = 0, 0 blocks, skip the launch, alternative formula, signed int, unsigned type
3.10 Part B: How big should each block be?
This section recommends a practical block size and explains its relationship to warp utilization and bounds checking.
- Block size and warp utilization: Explain why 256 threads per block is a useful starting choice and when to reconsider it.
Why it matters: A warp-aligned default avoids wasting hardware slots without distracting you from writing correct code. - Extra threads and bounds: Describe how the 10,000-element launch accommodates a block size of 256 and handles extra threads.
Why it matters: Rounding up lets the launch cover the entire input while the bounds check handles threads with no element to process.
Key terms: threadsPerBlock, warps, block size, profiling, threads per block, blocks, bounds check
3.11 The multi-block kernel recipe
This section brings together the multi-block recipe for processing arrays and explains the role of explicit synchronization.
- Multi-block array kernel recipe: Apply the four-step recipe to launch a kernel that processes a one-dimensional array.
Why it matters: This recipe lets you cover arrays of different sizes without leaving elements out or working past the end. - Repeated local index bug: Explain the faulty indexing behind a kernel that repeatedly modifies the first block’s elements while leaving later elements unchanged.
Why it matters: Recognizing repeated local indices helps you diagnose a multi-block kernel that only touches the start of an array. - Explicit kernel completion synchronization: Explain why the example explicitly synchronizes after the kernel and when that call is useful or costly.
Why it matters: Understanding the explicit wait helps you make kernel completion visible without adding unnecessary performance costs.
Key terms: block size, ceiling division, global index, bounds check, threadsPerBlock, numBlocks, threadIdx.x, blockIdx.x, blockDim.x, cudaDeviceSynchronize()
3.12 Common mistakes
This section previews three multi-block addressing mistakes and a symptom-first troubleshooting checklist.
3.13 Mistake 1: Using threadIdx directly
This section diagnoses overlapping array accesses caused by using a block-local thread index and explains how a global index fixes them.
- Thread index overlap: Explain why using threadIdx.x directly produces incorrect results with multiple blocks, how to recognize the problem, and how a global index fixes it.
Why it matters: Recognizing this mistake helps you diagnose wrong output that produces no crash or error message.
Key terms: threadIdx.x, blockDim.x, block offset, global index, index overlap
3.14 Mistake 2: Truncating instead of rounding up
This section explains why truncating division can leave the tail of an array unprocessed and how ceiling division fixes the block count.
- Rounding Up the Block Count: Explain why integer division can launch too few blocks when the array size is not a multiple of the block size, and describe how ceiling division fixes it.
Why it matters: A kernel can appear to run successfully while silently leaving the last elements unchanged.
Key terms: integer division, ceiling division, threadsPerBlock, block size
3.15 Mistake 3: Swapping the grid and block sizes
This section explains how swapping CUDA launch dimensions can cause failures or inefficient execution and how to correct their order.
- Grid and block argument order: Explain the correct order of CUDA launch arguments and the consequences of swapping them.
Why it matters: The wrong argument order can hide behind correct-looking results until a larger input causes the launch to fail.
Key terms: number of blocks, threads per block, cudaGetLastError(), warp
3.16 Summary
Where this is used later
- Introducing shared memory (chapter 11) · Develops the shared memory that threads within a block can use.
- Where threads share data: shared memory (chapter 5) · The distinction between threads and blocks.
- The barrier: __syncthreads() (chapter 5) · The relationship between blocks and threads.
- Why naive parallelism fails (chapter 5) · The limitations of naive parallel work become important when values must be combined.
- Tree reduction: from concept to code (chapter 5) · Reduction develops techniques for combining values in parallel.
- The last piece: atomic operations (chapter 5) · Combining block results requires additional techniques.
- Breaking it down, line by line (chapter 4) · A thread index alone does not uniquely identify a thread across blocks.
- The shared-variable trap: many threads, one counter (chapter 5) · The shared-variable trap explores competing updates to one counter.
Concepts explained in this chapter
- global indexA global index identifies a thread's corresponding element across all blocks in a CUDA grid.
- ceiling divisionCeiling division is division rounded upward to the smallest integer greater than or equal to the exact quotient.
- block sizeThreads per block is the configured number of GPU threads assigned to each CUDA thread block in a kernel launch.
- bounds checkingA bounds check is a conditional test that verifies an index lies within a data structure's valid range before access or computation.
- thread blockA block is a CUDA launch hierarchy unit whose threads are grouped and scheduled for execution on GPU hardware.
- thread limitA thread limit is the maximum number of threads that can be assigned to one CUDA block.
- CUDA execution hierarchyThe CUDA execution hierarchy consisting of grids, blocks, and threads.
- gridA grid is a CUDA launch hierarchy containing the thread blocks of a kernel execution.
- threadA thread is a sequence of instructions that performs a unit of computation.
- memory hierarchyA memory hierarchy is an arrangement of storage levels with differing capacities, access latencies, and bandwidths.
- block dimensionBlock dimensions are the configured extents of a CUDA thread block along its coordinate axes.
- CUDA kernelA CUDA kernel is a function launched on a GPU to execute concurrently across many GPU threads.
- cudaDeviceSynchronizecudaDeviceSynchronize is a CUDA runtime API function that blocks the host until previously issued work on the device has completed.
- kernel launchA kernel launch starts execution of a GPU kernel across a specified number of threads.
- streaming multiprocessorA streaming multiprocessor is a GPU hardware unit that schedules and executes groups of GPU threads using its execution resources.
- integer divisionInteger division is division whose result is an integer obtained by truncating the fractional part of the quotient.
- GPU memoryDevice memory is memory located on a GPU and accessible to GPU computations.
- hardware registerA GPU register is a tiny, fast storage location private to an individual thread.
- shared memoryShared memory is a fast, explicitly managed memory space accessible to threads within a GPU thread block.
- cudaGetLastErrorcudaGetLastError is a CUDA runtime API function that returns the latest CUDA error status for the calling host thread.
- out-of-bounds array accessAn out-of-bounds access is an attempt to read or write outside the valid range of a data structure or memory allocation.
- one thread per elementOne thread per element is a data-mapping approach that assigns one parallel thread to each element of an input collection.
- threadIdx.xthreadIdx.x is a CUDA-provided per-thread coordinate identifying a thread's position along the x dimension within its block.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
