liveBook · Manning

SPMD

SPMD is a parallel programming model in which multiple program instances execute the same code while operating on different data.

How 3 Manning books explain it

Deep Learning with JAX

A parallel programming model in which the same program runs simultaneously on multiple devices while each device processes different data.

How it is used
Deploys identical computations across multiple accelerators so each device processes a separate data shard.
How deep it goes
Explains pmap() compilation, replication, input layout, nesting with vmap(), and communication overhead, but defers tensor sharding.
Examples
Demonstrates vector dot products across eight devices and data-parallel MNIST image-classification training.
What you can do after
Apply it via JAX pmap() and vmap() to distribute batched computations across devices and verify equivalent results.

Read how Deep Learning with JAX explains SPMD →

GPU Programming with Triton

The SPMD model runs the same program code across multiple program instances while each instance operates on a different portion of the data.

How it is used
Maps one program across independent data chunks to express GPU data parallelism.
How deep it goes
Defines the pattern, distinguishes it from SIMT, and explains Triton’s block mapping; block-size performance tuning is deferred.
Examples
Uses vector addition, assigning contiguous input chunks to instances that produce corresponding output chunks.
What you can do after
Apply via Triton by computing a program ID, deriving its data offsets, and launching instances to cover the input.

Read how GPU Programming with Triton explains SPMD →

Grokking GPU Programming

SPMD is a programming model in which one program is executed by many parallel threads, with each thread operating on different data.

How it is used
Uses one kernel across many threads to apply the same operation to different data elements in GPU computing.
How deep it goes
Defines SPMD and distinguishes it from SIMT, while deferring details of hardware execution to a later discussion.
Examples
Shows a four-thread CUDA kernel that prints only when threadIdx.x equals 2.
What you can do after
Apply it by writing one kernel for many threads, with each thread working on its own data.

Read how Grokking GPU Programming explains SPMD →

What to know first

Related concepts

Questions about SPMD

What is SPMD?

SPMD is a parallel programming model in which multiple program instances execute the same code while operating on different data.

Which Manning books explain SPMD?

Deep Learning with JAX, GPU Programming with Triton and Grokking GPU Programming.

What should I know before learning SPMD?

Manning books that teach SPMD usually introduce tensor and warp first.

What is the difference between SPMD and SIMT?

SPMD is a parallel programming model in which multiple program instances execute the same code while operating on different data. SIMT is a processor execution model in which groups of threads execute a common instruction stream while operating on separate data.

What is SPMD a kind of?

SPMD is a kind of block-level programming. Block-level programming expresses a GPU computation for an entire block or tile rather than for an individual thread.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.