liveBook · Manning

Data parallelism

As explained in GPU Programming with Triton

Data parallelism applies the same computation independently to multiple partitions of data.

Data parallelism applies the same computation independently across data partitions. Element-wise tensor operations are a clear case: each output depends on one input, so workers need no coordination. section 1.3.2

GPUs suit this work by pursuing throughput: many simple arithmetic units process tasks at once, unlike CPUs, which favor completing individual tasks quickly. In Triton, host code divides work into independent tiles, and the device computes many tiles in parallel. section 1.3.2 section 1.5

Figure 1.5 GPUs dedicate more of their area to compute and less to control and caching.
Growth in training data

This figure contrasts GPU and CPU chip-area priorities, illustrating why GPUs favor throughput-oriented work. section 1.3.2

How GPU Programming with Triton covers it

How deep it goes
Defines independent-output work and its GPU motivation, but does not develop a kernel implementation.
Examples
Uses element-wise tensor arithmetic, where each output depends on one input and requires no worker coordination.
What you can do after
Understand why GPUs suit independent element-wise workloads and how Triton maps work across tiles.
How it connects
Connects device-side data parallelism with concurrency, parallelism, GPU throughput design, and Triton tiles.

Also explained in

Compare how 12 books explain data parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.