liveBook · Manning

SIMT

As explained in GPU Programming with Triton

The Single Instruction, Multiple Threads (SIMT) execution model runs the same instruction across multiple threads operating on separate data.

Single Instruction, Multiple Threads (SIMT) is a GPU execution model in which a group of threads normally executes the same instruction on separate data. Think of each thread as a worker handling its own tensor element while sharing an instruction stream with nearby workers. This differs from a CPU thread, which typically follows its own independent instruction sequence. section 1.3.3

The hardware issues instructions to groups called warps. On NVIDIA GPUs, a warp has 32 threads; one or more warps form a block that processes a tile of data. Threads need not remain in strict lockstep: branches can send them down different paths, and on GPUs since Volta, threads have individual program counters and can make progress independently. Threads within a warp can therefore race with one another. section 1.3.3

Triton exposes a higher-level, block-oriented SPMD model rather than requiring explicit coordination of individual threads. A program instance describes the computation for a tile, and a grid launches instances over the input. The compiler handles the internal threading and synchronization within each block. section 2.1.2

Figure 2.2 CUDA distributes work across many individual threads that each operate on a single element. Triton distributes work across blocks of elements, with each program instance processing one tile as a unit. This shift changes the granularity of parallelism and simplifies the programming model.
CH02 F02 Fartale

Compare CUDA's thread-level work distribution with Triton's block-level program instances. section 2.1.2

Independent element-wise work

For vector addition, each output element is the sum of the corresponding elements in two input vectors. Threads perform the same operation on different data, without needing to synchronize with one another. Such embarrassingly parallel work maps naturally to SIMT, though element-wise workloads can be limited by memory bandwidth. section 3.1.1

Divergence in conditional ReLU

Suppose a warp applies ReLU using an if/else and its inputs include both negative and positive values. The warp executes the paths serially, activating only the threads whose inputs match each path. If each path takes 10 cycles, the divergent warp takes 20 cycles in this simplified example. When every thread takes the same path, this divergence penalty does not occur. section 3.4.1

Figure 3.6 When threads in the same warp evaluate an if/else condition differently, the GPU must execute both branches serially. During each branch, only a subset of lanes is active while the others are idle, reducing effective parallelism and throughput.
CH03 F06 Fartale

See how inactive lanes reduce effective parallelism during divergent branches. section 3.4.1

How GPU Programming with Triton covers it

How deep it goes
Explains warp instruction issue, divergent branches, and independent thread progress on GPUs since Volta.
Examples
Uses vector addition and conditional ReLU with mixed-sign inputs.
What you can do after
Evaluate how thread grouping and divergent branches affect GPU execution.
How it connects
Connects SIMT with SPMD, warp divergence, and embarrassingly parallel workloads.

Also explained in

Compare how 3 books explain SIMT →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.