liveBook · Manning

Warp

As explained in GPU Programming with Triton

A warp is a hardware group of threads that an NVIDIA streaming multiprocessor schedules and issues together.

A warp is a group of 32 threads that an NVIDIA streaming multiprocessor (SM) schedules and issues instructions to together. In the single-instruction, multiple-threads (SIMT) model, the threads normally execute the same instruction on their own pieces of data. section 1.5 section 3.4.1

Threads form warps, warps belong to blocks, and blocks make up a grid. Triton kernels are written from the perspective of a program instance, corresponding to a CUDA block; the compiler manages the underlying threads and warps. section 1.3.3 section 1.5

Counting warps

A block of 128 threads contains 128 / 32 = 4 warps. A grid of 400 such blocks contains 1,600 warps in total. section 1.6

When threads in a warp take different branches, warp divergence can serialize those paths. For a conditional ReLU on mixed positive and negative inputs, one path runs with only its matching threads active, then the other path runs. If every thread takes the same branch, there is no divergence. section 3.4.1

Figure 3.6 When threads in the same warp evaluate an if/else condition differently, the GPU must execute both branches serially. During each branch, only a subset of lanes is active while the others are idle, reducing effective parallelism and throughput.
CH03 F06 Fartale

The diagram shows how divergent branches run serially with different subsets of a warp active. section 3.4.1

Occupancy is the ratio of active warps on an SM to the maximum it supports. More active warps can help hide memory latency, but increasing register use can reduce the number that fit. Higher occupancy alone does not guarantee better performance; avoiding register spills also matters. section 3.2.1

Tuning a Triton program instance

The num_warps launch parameter controls how many warps execute each program instance, without changing the tile’s logical shape or correctness. Too few can leave tile work under-parallelized; too many can reduce occupancy. A fused softmax example uses 4 warps for smaller tiles and 8 when BLOCK_SIZE reaches 512. section 4.2.3

How GPU Programming with Triton covers it

How deep it goes
Defines warp execution and works through divergence, occupancy, and launch-parameter tradeoffs.
Examples
Uses a thread-counting exercise, conditional ReLU, and a fused softmax launch to illustrate warps.
What you can do after
Operate/configure Triton’s num_warps parameter using tile size and register pressure as tuning considerations.
How it connects
Links SIMT execution to warp divergence, occupancy, register pressure, and latency hiding.

Also explained in

Compare how 3 books explain warp →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.