4 The Map pattern
This chapter covers
- The map pattern: a universal parallel primitive
- One thread per element, scaled across multiple blocks
- The Parallel Checklist: four questions that identify any parallel pattern
- Vector addition: your first two-input map
- Element-wise operations on arrays, from scalar multiplication to a neural network’s ReLU
You’ve already written a map kernel. In Chapter 3, when you doubled every element of an array (each thread took one element, multiplied it by two, and stored the result), that was a map. You just didn’t have the name for it yet.
The map pattern in one sentence: apply the same operation to many elements, each processed independently, none affecting any other. What makes it special is that every output depends only on same-index inputs, never on a neighbor’s: a thread may read more than one input, like both A[i] and B[i] for vector addition, as long as every input sits at its own index i. Thread 0 reads A[0] and B[0] and writes C[0]; thread 1 reads A[1] and B[1] and writes C[1]. (In Chapter 3 the doubling kernel wrote back into the same array, data[i] = data[i] * 2; here we read from A and B and write to a separate output array C. Both are maps: what matters is that thread i only ever touches index i, whether the output lands in the same array or a new one.) No thread ever needs to look at what any other thread is doing. No waiting, no coordination, no conflict. This is embarrassingly parallel at its purest, the best kind of problem to have.