liveBook · Manning

Data parallelism

As explained in Grokking GPU Programming

Data parallelism performs the same operation concurrently on multiple pieces of data.

Imagine doubling every number in an array. Each result depends only on its own input, so the same operation can be performed on many elements at once. This is data parallelism: applying the same operation concurrently to different pieces of data. section 1.4 section 1.3

Doubling an array

A simple sequential loop doubles 1,000 elements one at a time. A GPU-style approach assigns each element to a thread; every thread runs the same operation on its own element. The work is data parallel even if it is run sequentially. In practice, GPU threads are scheduled in groups, and a vectorized CPU can also process several elements at once. section 1.4

Figure 1.2 Parallel processing: every element at once. A GPU assigns one thread per element, doubling all 1,000 values in parallel; the hardware schedules those 1,000 threads onto its many CUDA cores.

See how threads are assigned to individual array elements in the parallel version. section 1.4

Data parallelism differs from task parallelism, where different operations run at the same time—for example, compressing one file while encrypting another. GPUs are especially suited to large amounts of independent, identical work; small inputs, branching, or step-by-step dependencies can favor a CPU. section 1.3 section 1.10

In CUDA, one kernel can run across many threads, each working on a different piece of data. This way of writing the program is called single program, multiple data (SPMD). A thread uses its index to find its work; for example, threadIdx.x identifies a thread within a one-dimensional block. section 2.7 section 2.24

Keep correctness in view when applying the pattern:

  • Check dependencies first. If one element needs another element’s result before it can be processed, the work cannot all start independently. section 1.10
  • Check array bounds. If extra threads are launched, a guard such as if (idx < n) prevents them from accessing elements outside the data. section 2.24
  • Synchronize before relying on results: a CUDA kernel launch is asynchronous, so the CPU can move on before the GPU finishes. section 2.24

More parallel workers do not guarantee unlimited speedup: sequential portions still limit the result. Start by finding independent work, then choose how to distribute it. section 1.10

How Grokking GPU Programming covers it

How deep it goes
Explains sequential and parallel execution, thread indexing, bounds checks, and synchronization; previews multidimensional layouts.
Examples
Illustrates the approach with array doubling and image shading.
What you can do after
Apply it via CUDA by assigning data to threads, checking bounds, and synchronizing before using results.
How it connects
Distinguishes task parallelism and relates data parallelism to SPMD and SIMD.

Also explained in

Compare how 12 books explain data parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.