SPMD
SPMD is a parallel programming model in which multiple program instances execute the same code while operating on different data.
Deep Learning with JAX
A parallel programming model in which the same program runs simultaneously on multiple devices while each device processes different data.
- How it is used
- Deploys identical computations across multiple accelerators so each device processes a separate data shard.
- How deep it goes
- Explains pmap() compilation, replication, input layout, nesting with vmap(), and communication overhead, but defers tensor sharding.
- Examples
- Demonstrates vector dot products across eight devices and data-parallel MNIST image-classification training.
- What you can do after
- Apply it via JAX pmap() and vmap() to distribute batched computations across devices and verify equivalent results.
GPU Programming with Triton
The SPMD model runs the same program code across multiple program instances while each instance operates on a different portion of the data.
- How it is used
- Maps one program across independent data chunks to express GPU data parallelism.
- How deep it goes
- Defines the pattern, distinguishes it from SIMT, and explains Triton’s block mapping; block-size performance tuning is deferred.
- Examples
- Uses vector addition, assigning contiguous input chunks to instances that produce corresponding output chunks.
- What you can do after
- Apply via Triton by computing a program ID, deriving its data offsets, and launching instances to cover the input.
Grokking GPU Programming
SPMD is a programming model in which one program is executed by many parallel threads, with each thread operating on different data.
- How it is used
- Uses one kernel across many threads to apply the same operation to different data elements in GPU computing.
- How deep it goes
- Defines SPMD and distinguishes it from SIMT, while deferring details of hardware execution to a later discussion.
- Examples
- Shows a four-thread CUDA kernel that prints only when threadIdx.x equals 2.
- What you can do after
- Apply it by writing one kernel for many threads, with each thread working on its own data.
What to know first
- tensorA tensor is a multidimensional numerical array whose values are arranged along indexed dimensions and can be manipulated through mathematical computation.
- warpA warp is a group of GPU threads scheduled and issued together for execution, typically processing the same instruction on different data.
Related concepts
- SIMTcontrasts with · SIMT is a processor execution model in which groups of threads execute a common instruction stream while operating on separate data.
- Tritonused by · Triton is a programming language and compiler for writing and compiling high-performance GPU kernels.
- data parallelismproduces · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- parallel mapequivalent to · A parallel map applies a computation concurrently to multiple inputs and produces corresponding results.
- block-level programmingkind of · Block-level programming expresses a GPU computation for an entire block or tile rather than for an individual thread.
- tl.program_iduses · tl.program_id is the Triton API call that returns the current program instance's ID along a specified launch-grid axis.
Questions about SPMD
What is SPMD?
SPMD is a parallel programming model in which multiple program instances execute the same code while operating on different data.
Which Manning books explain SPMD?
Deep Learning with JAX, GPU Programming with Triton and Grokking GPU Programming.
What should I know before learning SPMD?
Manning books that teach SPMD usually introduce tensor and warp first.
What is the difference between SPMD and SIMT?
SPMD is a parallel programming model in which multiple program instances execute the same code while operating on different data. SIMT is a processor execution model in which groups of threads execute a common instruction stream while operating on separate data.
What is SPMD a kind of?
SPMD is a kind of block-level programming. Block-level programming expresses a GPU computation for an entire block or tile rather than for an individual thread.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


