liveBook · Manning

Warp

As explained in Grokking GPU Programming

A warp is a GPU execution group whose threads typically execute the same instruction concurrently on different data.

When a GPU doubles an array, you can give each element its own thread. The threads do not all run at once: the hardware groups them into warps and schedules those groups. This is one reason the parallel job still takes time, even when its elements are independent. section 1.4

On an NVIDIA GPU, a warp is a group of 32 threads that normally execute the same instruction at the same time, each on its own data. Think of a marching band playing the same note together. This hardware execution model is called SIMT, or single instruction, multiple threads. You write and launch threads; the hardware packs them into warps. section 2.8 section 14.1

Packing a block

A block of 256 threads forms eight warps: threads 0–31 are in warp 0, threads 32–63 in warp 1, and so on. A thread stays in its assigned warp, and a warp never contains threads from different blocks. If a block has only eight threads, it still occupies a warp, leaving 24 lanes idle; a lane is a thread’s slot in the warp. section 14.1 section 2.8 section 14.12

Figure 14.1 A block of 256 threads is divided into eight warps of 32 consecutive threads each: warp 0 holds threads 0–31, warp 1 holds threads 32–63, and so on up to warp 7 (threads 224–255). The grouping is fixed: a thread never moves to another warp.

See how consecutive threads in a 256-thread block are grouped into eight warps. section 14.1

Blocks are assigned to streaming multiprocessors (SMs), which run their warps. An SM can keep many warps resident and switch to another when one stalls on memory. The ratio of active warps to the maximum an SM can hold is occupancy; more occupancy can help hide latency, but does not always improve performance. section 2.8 section 14.12

If threads within a warp take different branches, the warp runs the paths in turn. This branch divergence can slow a kernel; different warps taking different branches do not have the same cost. Keeping branches uniform within a warp or sorting data so nearby threads take the same path can help, but sorting must pay for its own cost. section 14.1 section 14.12

Do not use a warp’s usual togetherness as synchronization: since Volta, lockstep is not guaranteed. Likewise, do not assume warps in a block run in a fixed order; that mistake can cause intermittent failures. Use appropriate synchronization when threads depend on one another, and measure performance rather than assuming more resident warps are always faster. section 14.12 section 14.9

How Grokking GPU Programming covers it

How deep it goes
Explains warp grouping, scheduling, divergence, occupancy, and synchronization pitfalls, with practical tuning considerations.
Examples
Uses array doubling, a 32-musician marching band, and a 256-thread block split into eight warps.
What it assumes
Assumes basic C for the introductory CUDA program while introducing hardware thread grouping in plain language.
What you can do after
Diagnose performance and correctness problems using warp grouping, divergence, occupancy, and synchronization constraints.
How it connects
Connects warps to SIMT, branch divergence, occupancy, memory coalescing, shared memory bank conflicts, and constant memory broadcasts.

Also explained in

Compare how 3 books explain warp →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.