liveBook · Manning

Chapter 1: Why parallel programming?

What you will learn in chapter 1 of Grokking GPU Programming

This introduction explains why automatic hardware speedups gave way to multicore processors and identifies GPUs as a large source of processing units.

After this chapter you will be able to

Section by section

Introduction

This introduction explains why automatic hardware speedups gave way to multicore processors and identifies GPUs as a large source of processing units.

Key terms: free lunch, cores, GPU, processing units

1.1 The end of the “Free Lunch”

This section explains the trends that made processors faster, why heat ended easy clock-speed gains, and an alternative based on working together.

Key terms: Moore’s Law, transistors, power density, clock speeds, base clock speeds, throttle, power wall, grading room

1.2 The parallel computing revolution

This section introduces the shift toward using many processors and explains why more cores alone do not guarantee faster programs.

Key terms: CPU cores, GPU, CUDA cores, cores, split up, parallel programming

1.3 A family of parallel approaches

This section places GPUs within the broader family of parallel computing approaches and distinguishes data parallelism from task parallelism.

Key terms: parallel computing, processing unit, partial results, MPI, Message Passing Interface, MapReduce, OpenMP, Open Multi-Processing, SIMD, data parallelism

1.4 Sequential vs. parallel thinking

This section contrasts sequential and parallel processing through array doubling, practical speedup limits, and a reduction example.

Key terms: data parallelism, embarrassingly parallel, thread, scalar, SIMD, AVX, warps, speedup, parallel reduction, partial results

1.5 Caution: not everything can be parallelized

This section explains how dependencies limit parallelism and how Amdahl’s and Gustafson’s laws predict speedup under different workload assumptions.

Key terms: task, data dependency, tournament-style summation, Amdahl’s Law, sequential portion, critical path, speedup, Gustafson’s Law, sequential fraction, processors

1.6 Why GPUs became essential

This section explains why GPUs evolved from graphics hardware into general-purpose parallel processors and how their strengths and costs compare with CPUs.

Key terms: rendering, pixels, vertices, independent operations, CUDA, Compute Unified Device Architecture, graphics APIs, C-like syntax, host–device relationship, CPU

1.7 Why learn CUDA?

This section explains why learning CUDA helps diagnose performance problems and develops parallel-programming skills that transfer beyond GPUs.

Key terms: PyTorch, memory bandwidth, synchronization, CPU and GPU memory, parallel patterns, memory hierarchy, CPU threads, GPU threads, warps, CUDA

1.8 Your lab: swforces.com

This section introduces swforces.com as a browser-based CUDA practice lab and explains what its judge can and cannot assess.

Key terms: online judge, logical correctness, GPU performance, benchmark numbers

1.9 A preview of the patterns ahead

Introduces five recurring parallel patterns by describing the kinds of work and dependencies they represent.

Key terms: Map, Reduce, Scan, running totals, prefix sums, Histogram, bins, Stencil

1.10 Summary

Where this is used later

Concepts explained in this chapter

Chapter 2: Your first parallel program →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.