liveBook · Manning

SPMD

As explained in GPU Programming with Triton

The SPMD model runs the same program code across multiple program instances while each instance operates on a different portion of the data.

SPMD (Single Program, Multiple Data) applies the same program to multiple independent portions of data. Each instance runs the same code on a different slice. SPMD describes the programming pattern, not the hardware execution model. section 2.1.2

Triton uses blocked, or tiled, SPMD: each program instance handles a block of elements rather than a single element. A program ID determines which chunk it processes, so identical code covers consecutive chunks. section 2.1.2 section 2.2.4

Figure 2.2 CUDA distributes work across many individual threads that each operate on a single element. Triton distributes work across blocks of elements, with each program instance processing one tile as a unit. This shift changes the granularity of parallelism and simplifies the programming model.
CH02 F02 Fartale

Compare CUDA’s one-thread-per-element mapping with Triton’s one-program-per-tile mapping to see how the unit of work changes. section 2.1.2

Vector addition

For Z = X + Y, each instance adds corresponding chunks of X and Y and writes its results to Z. A grid of program instances covers the full vector. section 2.2

How GPU Programming with Triton covers it

How deep it goes
Defines the pattern, distinguishes it from SIMT, and explains Triton’s block mapping; block-size performance tuning is deferred.
Examples
Uses vector addition, assigning contiguous input chunks to instances that produce corresponding output chunks.
What it assumes
Starts with vector addition and defines host, device, and program-instance terminology while building the data mapping.
What you can do after
Apply via Triton by computing a program ID, deriving its data offsets, and launching instances to cover the input.
How it connects
Connects Triton and CUDA programming patterns to SIMT, warps, and block or tile processing.

Also explained in

Compare how 3 books explain SPMD →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.