chapter one

1 Why parallel programming?

 

This chapter covers

  • The shift from sequential to parallel computing
  • CUDA in the parallel programming landscape
  • The kinds of problem that can benefit from parallelization
  • How CPUs and GPUs divide the work: the host-device model
  • The tradeoff between latency and throughput

Suppose you need to sort a million numbers. You write a program, hit Run, and it takes ten seconds. Not bad. Next year, the same program runs in five seconds. You didn’t touch a single line of code. The hardware just got faster. For decades, that was how programming worked. You wrote it once, and time made it faster for free. Programmers of that era had a name for it: the free lunch.

Write some code. Wait a year. It runs twice as fast on new hardware. Those days existed. And then they ended.

— Inspired by Herb Sutter, “The Free Lunch Is Over” (2005)

Around 2005, that free ride stopped, and it hasn’t come back. Processors couldn’t get much faster on their own anymore. So chip makers tried a different trick: instead of one fast core, they put multiple cores on a single chip. But the biggest pool of cores in your machine is not in the CPU. Your GPU (graphics processing unit) carries thousands of tiny processing units: a mid-range laptop GPU such as the RTX 4060 packs 3,072 of them (NVIDIA calls them CUDA cores; AMD’s equivalent is stream processors; you’ll learn what CUDA is later in this chapter). Unless you’re gaming or editing video, most of that silicon sits idle.

The end of the “Free Lunch”

The parallel computing revolution

A family of parallel approaches

Sequential vs. parallel thinking

Caution: not everything can be parallelized

Why GPUs became essential

Why learn CUDA?

Your lab: swforces.com

A preview of the patterns ahead

Summary