liveBook · Manning

Parallelism

As explained in GPU Programming with Triton

Parallelism is the simultaneous execution of multiple tasks by multiple physical or computational resources.

Parallelism means executing multiple tasks at the same time using multiple physical or computational resources. It differs from concurrency: concurrency structures independent tasks so they can make overlapping progress, while parallelism runs tasks simultaneously. section 1.3.1

Coffee shop analogy

One cashier switching between several orders illustrates concurrency; several cashiers serving different customers simultaneously illustrate parallelism. On a GPU, many lightweight threads can execute independent work at once, provided the work is structured into independent tasks. section 1.3.1 section 1.5

Figure 1.4 In concurrency, a single worker (thread or core) switches between multiple tasks, giving the illusion of simultaneous progress. In parallelism, multiple workers execute separate tasks at the same instant.
CH01 F04 Fartale

Compare one worker switching among tasks with multiple workers executing separate tasks simultaneously. section 1.3.1

How GPU Programming with Triton covers it

How deep it goes
Defines parallelism, contrasts it with concurrency, and illustrates the distinction without detailing kernel implementation.
Examples
Uses a coffee shop with one or several cashiers and GPU threads processing independent work.
What you can do after
Understand how independent tasks can be structured for simultaneous execution on multiple resources.
How it connects
Connects parallelism with concurrency, independent tasks, data parallelism, and GPU execution.

Also explained in

Compare how 55 books explain parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.