chapter three

3 Scaling up: from one block to many

 

This chapter covers

  • Why a single block can’t handle large problems (the 1,024-thread limit)
  • CUDA’s three-level hierarchy: grids, blocks, and threads
  • The global index formula that gives every thread a unique ID
  • How to choose a block size, compute the block count, and bounds-check the extra threads
  • The four-step recipe that turns any 1D array problem into a correct kernel launch

In chapter 2, you learned how to launch a kernel, a function that runs on the GPU, and how each thread knows its own position using threadIdx.x. You processed arrays by giving each thread one element, and that worked well as long as the arrays were small enough for a single block to cover. But what happens when an array outgrows that block? In this chapter, “scaling up” means one specific move: going from a single block to many, so the same kernel can handle arrays far larger than any one block could hold.

Picture the grading room from chapter 1 again, but with a bigger stack: you’re a teacher who just collected 10,000 exams, and every one of them needs a grade. Grade them one by one, and you’ll be at it for weeks.

Now suppose you could open 40 classrooms, put 250 graders in each room, hand each room its own stack of exams, and say “Go!” All 40 rooms work at the same time, and weeks of grading shrink to roughly the time it takes to grade one exam.

What’s wrong with one block?

The three-level hierarchy: grids, blocks, and threads

Why the thread index alone isn’t enough

The global index formula

The global index as a student ID

Tracing through the numbers

Your first multi-block kernel

Launching the right number of threads

Part A: How many blocks do I need?

Part B: How big should each block be?

The multi-block kernel recipe

Common mistakes

Mistake 1: Using threadIdx directly

Mistake 2: Truncating instead of rounding up

Mistake 3: Swapping the grid and block sizes

Summary