chapter four

4 The Map pattern

 

This chapter covers

  • The map pattern, a universal parallel primitive
  • One thread per element, scaled across multiple blocks
  • The Parallel Checklist: four questions you’ll ask of every parallel pattern in this book
  • Your first two-input map with vector addition
  • Element-wise operations on arrays, from scalar multiplication to a neural network’s ReLU

You’ve already written a map kernel. In chapter 3, when you doubled every element of an array (each thread took one element, multiplied it by two, and stored the result), that was a map, and so was chapter 2’s squaring kernel. Chapter 1 called this pattern map; this chapter shows what makes it work.

The map pattern applies the same operation to many elements, which are each processed independently so that none affects any of the others. This means that no output depends on any other output, and in a pure map every output reads only same-index inputs, never a neighbor’s.

What makes a problem “mappable”?

Where you’ve seen Map before

One thread, one element

Map in action: vector addition

Thinking in parallel

Breaking it down, line by line

Map in action: scalar vector multiplication

Thinking in parallel

Breaking it down, line by line

Map in action: vector ReLU

Thinking in parallel

Breaking it down, line by line

Common mistakes

Mistake 1: Accessing another thread’s element

Mistake 2: Accidentally overwriting your input

Summary