liveBook · Manning

SPMD

As explained in Grokking GPU Programming

SPMD is a programming model in which one program is executed by many parallel threads, with each thread operating on different data.

SPMD means “single program, multiple data”: one kernel runs across many threads, and each thread works on a different slice of the data. Applying the same operation to different data elements is data parallelism, the foundation of GPU computing. section 2.7

SPMD describes how the code is written; SIMT describes how hardware runs the threads. Thread execution order is not guaranteed, so correctness should not depend on which thread runs first. section 2.7

#include <stdio.h>
#include <cuda_runtime.h>
 
__global__ void printIndexKernel() {
    int idx = threadIdx.x;     #A
    if (idx == 2) {    #B
        printf("Thread %d: Hello!\n", idx);    #C
    }
}
 
int main() {
    printIndexKernel<<<1, 4>>>();
    cudaDeviceSynchronize();
    return 0;
}

This kernel launches four threads, each reading its own threadIdx.x; only thread 2 prints, illustrating one program running across threads. section 2.7

How Grokking GPU Programming covers it

How deep it goes
Defines SPMD and distinguishes it from SIMT, while deferring details of hardware execution to a later discussion.
Examples
Shows a four-thread CUDA kernel that prints only when threadIdx.x equals 2.
What you can do after
Apply it by writing one kernel for many threads, with each thread working on its own data.
How it connects
Connects SPMD with CUDA, SIMT, data parallelism, and task parallelism.

Also explained in

Compare how 3 books explain SPMD →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.