When a GPU doubles an array, you can give each element its own thread. The threads do not all run at once: the hardware groups them into warps and schedules those groups. This is one reason the parallel job still takes time, even when its elements are independent. section 1.4
On an NVIDIA GPU, a warp is a group of 32 threads that normally execute the same instruction at the same time, each on its own data. Think of a marching band playing the same note together. This hardware execution model is called SIMT, or single instruction, multiple threads. You write and launch threads; the hardware packs them into warps. section 2.8 section 14.1
Packing a block
A block of 256 threads forms eight warps: threads 0–31 are in warp 0, threads 32–63 in warp 1, and so on. A thread stays in its assigned warp, and a warp never contains threads from different blocks. If a block has only eight threads, it still occupies a warp, leaving 24 lanes idle; a lane is a thread’s slot in the warp. section 14.1 section 2.8 section 14.12
Figure 14.1 A block of 256 threads is divided into eight warps of 32 consecutive threads each: warp 0 holds threads 0–31, warp 1 holds threads 32–63, and so on up to warp 7 (threads 224–255). The grouping is fixed: a thread never moves to another warp.

See how consecutive threads in a 256-thread block are grouped into eight warps. section 14.1
Blocks are assigned to streaming multiprocessors (SMs), which run their warps. An SM can keep many warps resident and switch to another when one stalls on memory. The ratio of active warps to the maximum an SM can hold is occupancy; more occupancy can help hide latency, but does not always improve performance. section 2.8 section 14.12
If threads within a warp take different branches, the warp runs the paths in turn. This branch divergence can slow a kernel; different warps taking different branches do not have the same cost. Keeping branches uniform within a warp or sorting data so nearby threads take the same path can help, but sorting must pay for its own cost. section 14.1 section 14.12
Do not use a warp’s usual togetherness as synchronization: since Volta, lockstep is not guaranteed. Likewise, do not assume warps in a block run in a fixed order; that mistake can cause intermittent failures. Use appropriate synchronization when threads depend on one another, and measure performance rather than assuming more resident warps are always faster. section 14.12 section 14.9


