A warp is a group of 32 threads that an NVIDIA streaming multiprocessor (SM) schedules and issues instructions to together. In the single-instruction, multiple-threads (SIMT) model, the threads normally execute the same instruction on their own pieces of data. section 1.5 section 3.4.1
Threads form warps, warps belong to blocks, and blocks make up a grid. Triton kernels are written from the perspective of a program instance, corresponding to a CUDA block; the compiler manages the underlying threads and warps. section 1.3.3 section 1.5
Counting warps
A block of 128 threads contains 128 / 32 = 4 warps. A grid of 400 such blocks contains 1,600 warps in total. section 1.6
When threads in a warp take different branches, warp divergence can serialize those paths. For a conditional ReLU on mixed positive and negative inputs, one path runs with only its matching threads active, then the other path runs. If every thread takes the same branch, there is no divergence. section 3.4.1
Figure 3.6 When threads in the same warp evaluate an if/else condition differently, the GPU must execute both branches serially. During each branch, only a subset of lanes is active while the others are idle, reducing effective parallelism and throughput.
The diagram shows how divergent branches run serially with different subsets of a warp active. section 3.4.1
Occupancy is the ratio of active warps on an SM to the maximum it supports. More active warps can help hide memory latency, but increasing register use can reduce the number that fit. Higher occupancy alone does not guarantee better performance; avoiding register spills also matters. section 3.2.1
Tuning a Triton program instance
The num_warps launch parameter controls how many warps execute each program instance, without changing the tile’s logical shape or correctness. Too few can leave tile work under-parallelized; too many can reduce occupancy. A fused softmax example uses 4 warps for smaller tiles and 8 when BLOCK_SIZE reaches 512. section 4.2.3


