Imagine doubling every number in an array. Each result depends only on its own input, so the same operation can be performed on many elements at once. This is data parallelism: applying the same operation concurrently to different pieces of data. section 1.4 section 1.3
Doubling an array
A simple sequential loop doubles 1,000 elements one at a time. A GPU-style approach assigns each element to a thread; every thread runs the same operation on its own element. The work is data parallel even if it is run sequentially. In practice, GPU threads are scheduled in groups, and a vectorized CPU can also process several elements at once. section 1.4
Figure 1.2 Parallel processing: every element at once. A GPU assigns one thread per element, doubling all 1,000 values in parallel; the hardware schedules those 1,000 threads onto its many CUDA cores.

See how threads are assigned to individual array elements in the parallel version. section 1.4
Data parallelism differs from task parallelism, where different operations run at the same time—for example, compressing one file while encrypting another. GPUs are especially suited to large amounts of independent, identical work; small inputs, branching, or step-by-step dependencies can favor a CPU. section 1.3 section 1.10
In CUDA, one kernel can run across many threads, each working on a different piece of data. This way of writing the program is called single program, multiple data (SPMD). A thread uses its index to find its work; for example, threadIdx.x identifies a thread within a one-dimensional block. section 2.7 section 2.24
Keep correctness in view when applying the pattern:
- Check dependencies first. If one element needs another element’s result before it can be processed, the work cannot all start independently. section 1.10
- Check array bounds. If extra threads are launched, a guard such as if (idx < n) prevents them from accessing elements outside the data. section 2.24
- Synchronize before relying on results: a CUDA kernel launch is asynchronous, so the CPU can move on before the GPU finishes. section 2.24
More parallel workers do not guarantee unlimited speedup: sequential portions still limit the result. Start by finding independent work, then choose how to distribute it. section 1.10











