chapter one
1 Why parallel programming?
This chapter covers
- Why making one processor faster hit a physical ceiling (the Power Wall)
- How the world shifted from sequential to parallel computing
- Why GPUs went from drawing pixels to powering AI
- How learning CUDA builds parallel-thinking skills that transfer to any framework
- How to recognize which problems can benefit from parallelization (and which can’t)
- How CUDA fits into a longer parallel programming tradition alongside MPI, OpenMP, and SIMD
Suppose you need to sort a million numbers. You write a program, hit Run, and it takes ten seconds. Not bad. Next year, the same program runs in five seconds. You didn’t touch a single line of code. The hardware just got faster. For decades, that was how programming worked. You wrote it once, and time made it faster for free. Programmers of that era had a name for it: the free lunch.
“Write some code. Wait a year. It runs twice as fast on new hardware. Those days existed. And then they ended.”
— Inspired by Herb Sutter, “The Free Lunch Is Over” (2005)