liveBook · Manning

Warp

A warp is a group of GPU threads scheduled and issued together for execution, typically processing the same instruction on different data.

How 3 Manning books explain it

CUDA for LLMs

A warp is a group of GPU threads scheduled to execute instructions together on a streaming multiprocessor.

How it is used
Maps GPU work to cooperative thread groups for memory access, reductions, and tensor-core tiles.
How deep it goes
Explains lane-level access and shuffle reductions, then traces warp cooperation through WMMA fragments and hardware-specific tile mapping.
Examples
Uses GEMV, fused LayerNorm, and tiled WMMA matrix multiplication to demonstrate warp cooperation.
What you can do after
Apply it via CUDA to map adjacent accesses and combine per-lane partial sums with warp shuffles.

Read how CUDA for LLMs explains warp →

GPU Programming with Triton

A warp is a hardware group of threads that an NVIDIA streaming multiprocessor schedules and issues together.

How it is used
Connects Triton program instances to hardware scheduling and performance considerations.
How deep it goes
Defines warp execution and works through divergence, occupancy, and launch-parameter tradeoffs.
Examples
Uses a thread-counting exercise, conditional ReLU, and a fused softmax launch to illustrate warps.
What you can do after
Operate/configure Triton’s num_warps parameter using tile size and register pressure as tuning considerations.

Read how GPU Programming with Triton explains warp →

Grokking GPU Programming

A warp is a GPU execution group whose threads typically execute the same instruction concurrently on different data.

How it is used
Explains how launched CUDA threads execute in groups and why that grouping affects kernel performance and correctness.
How deep it goes
Explains warp grouping, scheduling, divergence, occupancy, and synchronization pitfalls, with practical tuning considerations.
Examples
Uses array doubling, a 32-musician marching band, and a 256-thread block split into eight warps.
What you can do after
Diagnose performance and correctness problems using warp grouping, divergence, occupancy, and synchronization constraints.

Read how Grokking GPU Programming explains warp →

Also covered in

What to know first

Related concepts

Questions about warp

What is warp?

A warp is a group of GPU threads scheduled and issued together for execution, typically processing the same instruction on different data.

Which Manning books explain warp?

CUDA for LLMs, GPU Programming with Triton and Grokking GPU Programming.

What should I know before learning warp?

Manning books that teach warp usually introduce thread, hardware register, GPU memory and matrix-matrix multiplication first.

What is warp part of?

Warp is part of thread block. A thread block is a group of GPU threads organized to execute together and cooperate using shared resources.

What are the parts of warp?

Warp includes thread.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.