Chapter 1: Why parallel programming?
This introduction explains why automatic hardware speedups gave way to multicore processors and identifies GPUs as a large source of processing units.
After this chapter you will be able to
- The Free Lunch. Explain what the free lunch meant for existing programs and why it ended.
- Multicore Chips and GPUs. Describe how chip makers responded to slowing processor speedups and where GPUs fit into that picture.
- Sources of Faster Processors. Explain how growing transistor counts and lower power per transistor enabled faster processors without worsening cooling requirements.
- The Power Wall. Explain why CPU base clock speeds plateaued around 3–4 GHz and how overheating affects modern chips.
- Shift to Multiple Processors. Explain how using many processors addresses the slowdown in single-core speed improvements, and distinguish CPU cores from the GPU cores described here.
- Limits of Adding Cores. Explain why more cores do not automatically make a program faster, using the exam-grading example or a similar explanation.
- Parallel computing common idea. Explain what parallel computing means and how GPUs relate to other parallel approaches.
- Data and task parallelism. Distinguish data parallelism from task parallelism and explain which GPUs are built for.
- Sequential and parallel array processing. Explain how sequential and parallel approaches double an array, and why the work is embarrassingly parallel.
- Realistic parallel speedup limits. Explain why the ideal one-step GPU comparison overstates the practical speedup over a CPU.
- Parallel reduction and work. Explain how a tournament-style reduction sums numbers in parallel and distinguish elapsed rounds from total additions.
- Dependencies and available parallelism. Explain how dependencies determine which tasks can run together, using tournament summation or straightforward Fibonacci as an example.
- Sequential bottlenecks and critical path. Explain why sequential work limits speedup for a fixed problem, using the 10% example and the idea of a critical path.
- Fixed versus growing workloads. Distinguish Amdahl’s and Gustafson’s laws and apply their formulas to estimate speedup.
- Graphics Origins of GPUs. Explain how rendering graphics led to GPU designs that also benefit general-purpose computing.
- Host–Device Partnership. Describe the host–device relationship between the CPU and GPU.
- Latency and Throughput Tradeoffs. Distinguish latency from throughput and explain how CPU and GPU designs favor each.
- CPU and GPU Workload Fit. Apply the section's workload-fit criteria to distinguish jobs better suited to a GPU from those better suited to a CPU.
- GPU Benchmark Interpretation. Explain why a strong GPU matrix-multiplication benchmark may not predict the speedup of a complete one-off job.
- Low-level performance diagnosis. Explain how CUDA knowledge can help diagnose slow programs built with high-level libraries, using the image-processing pipeline as an example.
- Transferable parallel programming ideas. Explain which CUDA ideas carry over to other parallel frameworks and how the section contrasts CPU and GPU threads.
- Online judge capabilities and limits. Explain how to use swforces.com to practice CUDA, what its verdict checks, and where to look for performance information.
- Independent Map operations. Explain how Map applies the same operation independently to each element, using an example.
- Reduce and Scan outputs. Distinguish Reduce from Scan by describing what each produces from a collection of values.
- Histogram and Stencil patterns. Distinguish Histogram and Stencil by describing the work each performs and an example of either.
Section by section
Introduction
This introduction explains why automatic hardware speedups gave way to multicore processors and identifies GPUs as a large source of processing units.
- The Free Lunch: Explain what the free lunch meant for existing programs and why it ended.
Why it matters: This explains why programmers could no longer rely on new hardware alone to speed up existing code. - Multicore Chips and GPUs: Describe how chip makers responded to slowing processor speedups and where GPUs fit into that picture.
Why it matters: This shift motivates using parallel processing resources rather than relying solely on a faster individual core.
Key terms: free lunch, cores, GPU, processing units
1.1 The end of the “Free Lunch”
This section explains the trends that made processors faster, why heat ended easy clock-speed gains, and an alternative based on working together.
- Sources of Faster Processors: Explain how growing transistor counts and lower power per transistor enabled faster processors without worsening cooling requirements.
Why it matters: These trends explain why processor performance used to improve so readily. - The Power Wall: Explain why CPU base clock speeds plateaued around 3–4 GHz and how overheating affects modern chips.
Why it matters: The power wall explains why simply raising CPU clock speed stopped being an easy route to more performance. - The Grading Room Alternative: Describe how the grading room illustrates getting more work done when one worker cannot work faster.
Why it matters: The grading room gives you a way to picture the alternative to making a single processor faster.
Key terms: Moore’s Law, transistors, power density, clock speeds, base clock speeds, throttle, power wall, grading room
1.2 The parallel computing revolution
This section introduces the shift toward using many processors and explains why more cores alone do not guarantee faster programs.
- Shift to Multiple Processors: Explain how using many processors addresses the slowdown in single-core speed improvements, and distinguish CPU cores from the GPU cores described here.
Why it matters: This shift explains why parallel programming matters for continuing to speed up programs. - Limits of Adding Cores: Explain why more cores do not automatically make a program faster, using the exam-grading example or a similar explanation.
Why it matters: It helps you recognize why adding processors is not a substitute for organizing work for parallel execution.
Key terms: CPU cores, GPU, CUDA cores, cores, split up, parallel programming
1.3 A family of parallel approaches
This section places GPUs within the broader family of parallel computing approaches and distinguishes data parallelism from task parallelism.
- Parallel computing common idea: Explain what parallel computing means and how GPUs relate to other parallel approaches.
Why it matters: This common idea connects GPU programming to other ways of solving problems in parallel. - Parallel approaches across scales: Describe the parallel approaches presented at cluster, shared-memory machine, and single-core scales.
Why it matters: Recognizing these approaches helps you see parallel computing beyond GPUs. - Data and task parallelism: Distinguish data parallelism from task parallelism and explain which GPUs are built for.
Why it matters: The distinction clarifies the kind of parallel work GPUs and this book emphasize.
Key terms: parallel computing, processing unit, partial results, MPI, Message Passing Interface, MapReduce, OpenMP, Open Multi-Processing, SIMD, data parallelism
1.4 Sequential vs. parallel thinking
This section contrasts sequential and parallel processing through array doubling, practical speedup limits, and a reduction example.
- Sequential and parallel array processing: Explain how sequential and parallel approaches double an array, and why the work is embarrassingly parallel.
Why it matters: Recognizing independent elementwise work is a starting point for thinking in parallel. - Realistic parallel speedup limits: Explain why the ideal one-step GPU comparison overstates the practical speedup over a CPU.
Why it matters: Idealized step counts help build intuition but can mislead you about actual speedup. - Parallel reduction and work: Explain how a tournament-style reduction sums numbers in parallel and distinguish elapsed rounds from total additions.
Why it matters: Parallel thinking can reduce elapsed time even when a problem is not embarrassingly parallel. - Fixed core step calculation: Apply the exercise's assumptions to find the minimum steps needed for eight cores to process 1,000 elements.
Why it matters: This calculation connects the number of simultaneous workers to the time needed for independent work.
Key terms: data parallelism, embarrassingly parallel, thread, scalar, SIMD, AVX, warps, speedup, parallel reduction, partial results
1.5 Caution: not everything can be parallelized
This section explains how dependencies limit parallelism and how Amdahl’s and Gustafson’s laws predict speedup under different workload assumptions.
- Dependencies and available parallelism: Explain how dependencies determine which tasks can run together, using tournament summation or straightforward Fibonacci as an example.
Why it matters: Identifying dependencies helps you see which parts of a problem can actually run at the same time. - Sequential bottlenecks and critical path: Explain why sequential work limits speedup for a fixed problem, using the 10% example and the idea of a critical path.
Why it matters: The sequential bottleneck sets expectations for how much extra hardware can help. - Fixed versus growing workloads: Distinguish Amdahl’s and Gustafson’s laws and apply their formulas to estimate speedup.
Why it matters: The workload assumption changes the speedup you should expect from additional processors.
Key terms: task, data dependency, tournament-style summation, Amdahl’s Law, sequential portion, critical path, speedup, Gustafson’s Law, sequential fraction, processors
1.6 Why GPUs became essential
This section explains why GPUs evolved from graphics hardware into general-purpose parallel processors and how their strengths and costs compare with CPUs.
- Graphics Origins of GPUs: Explain how rendering graphics led to GPU designs that also benefit general-purpose computing.
Why it matters: The graphics origin explains why GPUs suit some general-purpose problems better than others. - CUDA and General Computing: Describe how CUDA changed the way programmers could write general-purpose GPU code.
Why it matters: CUDA made GPU computing accessible without disguising ordinary calculations as graphics. - Host–Device Partnership: Describe the host–device relationship between the CPU and GPU.
Why it matters: GPU programs depend on the CPU and GPU cooperating rather than one replacing the other. - Latency and Throughput Tradeoffs: Distinguish latency from throughput and explain how CPU and GPU designs favor each.
Why it matters: The different design goals explain why adding CPU cores alone does not create the same tradeoff as a GPU. - CPU and GPU Workload Fit: Apply the section's workload-fit criteria to distinguish jobs better suited to a GPU from those better suited to a CPU.
Why it matters: Choosing the right processor starts with the shape of the work, not the assumption that a GPU is always faster. - GPU Benchmark Interpretation: Explain why a strong GPU matrix-multiplication benchmark may not predict the speedup of a complete one-off job.
Why it matters: End-to-end costs can turn an impressive raw GPU speedup into little or no practical gain.
Key terms: rendering, pixels, vertices, independent operations, CUDA, Compute Unified Device Architecture, graphics APIs, C-like syntax, host–device relationship, CPU
1.7 Why learn CUDA?
This section explains why learning CUDA helps diagnose performance problems and develops parallel-programming skills that transfer beyond GPUs.
- Low-level performance diagnosis: Explain how CUDA knowledge can help diagnose slow programs built with high-level libraries, using the image-processing pipeline as an example.
Why it matters: Low-level knowledge gives you useful questions to ask when a convenient library runs slowly. - Transferable parallel programming ideas: Explain which CUDA ideas carry over to other parallel frameworks and how the section contrasts CPU and GPU threads.
Why it matters: Learning to organize parallel work and avoid data interference remains useful on different hardware. - Real-world CUDA applications: Describe examples of important technologies that use CUDA and what the GPUs do in them.
Why it matters: These examples show why GPU programming matters in real applications.
Key terms: PyTorch, memory bandwidth, synchronization, CPU and GPU memory, parallel patterns, memory hierarchy, CPU threads, GPU threads, warps, CUDA
1.8 Your lab: swforces.com
This section introduces swforces.com as a browser-based CUDA practice lab and explains what its judge can and cannot assess.
- Online judge capabilities and limits: Explain how to use swforces.com to practice CUDA, what its verdict checks, and where to look for performance information.
Why it matters: This helps you practice CUDA while interpreting the judge’s verdict correctly.
Key terms: online judge, logical correctness, GPU performance, benchmark numbers
1.9 A preview of the patterns ahead
Introduces five recurring parallel patterns by describing the kinds of work and dependencies they represent.
- Independent Map operations: Explain how Map applies the same operation independently to each element, using an example.
Why it matters: Recognizing independent elementwise work helps you identify a natural parallel pattern. - Reduce and Scan outputs: Distinguish Reduce from Scan by describing what each produces from a collection of values.
Why it matters: The desired output distinguishes a single combined result from a running result for each position. - Histogram and Stencil patterns: Distinguish Histogram and Stencil by describing the work each performs and an example of either.
Why it matters: Counting categories and computing from neighbors are different kinds of parallel work.
Key terms: Map, Reduce, Scan, running totals, prefix sums, Histogram, bins, Stencil
1.10 Summary
Where this is used later
- Moving data: CPU to GPU and back (chapter 2) · Provides the grading-room setting used to illustrate moving data between CPU and GPU.
- Meet threadIdx (chapter 2) · The distinction between data and task parallelism.
- How the hardware runs your threads: warps and SMs (chapter 2) · The earlier introduction to parallel approaches.
- Where you’ve seen Map before (chapter 4) · The family of parallel approaches that includes SIMD, OpenMP, and MapReduce.
- What makes a problem “mappable”? (chapter 4) · Recognizing the same operation across independent data helps identify a mappable problem.
- One thread, one element (chapter 4) · Develops the one-thread-per-element approach.
- Mistake 3: Missing bounds check (chapter 2) · The distinction between sequential and parallel thinking.
- Tree reduction: from concept to code (chapter 5) · Develops the tree-reduction approach from concept to code.
Concepts explained in this chapter
- parallelismParallel computing is the execution of computational work simultaneously across multiple processing units.
- CUDACUDA is NVIDIA's parallel computing platform and programming model for executing general-purpose computations on GPUs.
- GPUA GPU is a processor designed to execute many operations concurrently, especially data-parallel operations.
- speedupSpeedup is the ratio of a computation's sequential execution time to its parallel execution time.
- reduction operationThe reduction pattern combines many input values into one result using an associative operation, often by combining partial results in parallel.
- CPUA CPU is a general-purpose processor that executes program instructions.
- data parallelismData parallelism performs the same operation concurrently on multiple pieces of data.
- latencyLatency is the time required to complete one operation or job.
- massively parallel computationMassively parallel computation performs the same or similar independent operations concurrently across a very large amount of data.
- throughputThroughput is the amount of work completed per unit of time or over an overall execution period.
- GPGPUGPU programming is the practice of writing software that executes computational work on graphics processing units.
- Amdahl’s LawAmdahl’s Law models the maximum speedup obtainable when the problem size is fixed and a fraction of the work remains sequential.
- Gustafson's lawGustafson’s Law models scalable speedup by allowing the problem size to grow as more processors are added.
- data dependencyA data dependency occurs when one computation requires a value produced or modified by another computation.
- scanThe scan pattern computes cumulative results for the prefixes of an ordered sequence.
- embarrassingly parallelAn embarrassingly parallel computation consists of independent work items that can be parallelized with little or no coordination.
- sequential processingSequential execution performs computational steps one at a time rather than concurrently.
- convolutionConvolution is an operation that combines input values with corresponding kernel weights over a neighborhood to compute outputs.
- histogramA histogram is a collection of counts recording how many values fall into each category or bin.
- mapThe map pattern applies an operation independently to each element of a collection.
- SIMD hardwareA SIMD/AVX unit is processor hardware that applies one vector instruction to multiple data elements at once.
- threadA thread is a sequence of instructions that performs a unit of computation.
- warpA warp is a GPU execution group whose threads typically execute the same instruction concurrently on different data.
- critical pathThe critical path is the longest chain of dependent steps that must execute sequentially in a computation.
- power wallThe power wall is a physical limit on increasing processor speed caused by power consumption and heat dissipation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
