liveBook · Manning

Block-level reduction

A block-level reduction combines values across threads within a GPU thread block into an aggregate result.

How 2 Manning books explain it

GPU Programming with Triton

A block-level reduction combines values within one GPU block or Triton program instance using on-chip resources and synchronization.

How it is used
Reduces per-instance values before global atomics to lessen contention on a shared output address.
How deep it goes
Explains tile-to-scalar aggregation through on-chip shuffles and shared memory, then compares atomic counts before and after reduction.
Examples
Compares 1,024 atomics per program across 256 instances with one atomic per instance.
What you can do after
Apply tl.sum before tl.atomic_add to combine each program’s contributions into one global-memory update.

Read how GPU Programming with Triton explains block-level reduction →

CUDA for LLMs

A block-level reduction combines values from threads within a GPU thread block into an aggregate result.

How it is used
Uses shared memory to extend warp-level reductions across a block in GEMV and parallel top-k kernels.
How deep it goes
Works through warp partials, shared-memory exchange, and synchronization, without detailing every helper implementation.
Examples
Uses a GEMV row sum and repeated top-k maximum-and-index reductions.
What you can do after
Apply it in CUDA kernels to combine thread-local values across a block.

Read how CUDA for LLMs explains block-level reduction →

Related concepts

Questions about block-level reduction

What is block-level reduction?

A block-level reduction combines values across threads within a GPU thread block into an aggregate result.

Which Manning books explain block-level reduction?

GPU Programming with Triton and CUDA for LLMs.

What is block-level reduction a kind of?

Block-level reduction is a kind of reduction operation. A reduction operation combines multiple input values according to a rule to produce fewer results, often a single aggregate.

What are the parts of block-level reduction?

Block-level reduction includes warp-level reduction and shared-memory reduction.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.