Block-level reduction
A block-level reduction combines values across threads within a GPU thread block into an aggregate result.
GPU Programming with Triton
A block-level reduction combines values within one GPU block or Triton program instance using on-chip resources and synchronization.
- How it is used
- Reduces per-instance values before global atomics to lessen contention on a shared output address.
- How deep it goes
- Explains tile-to-scalar aggregation through on-chip shuffles and shared memory, then compares atomic counts before and after reduction.
- Examples
- Compares 1,024 atomics per program across 256 instances with one atomic per instance.
- What you can do after
- Apply tl.sum before tl.atomic_add to combine each program’s contributions into one global-memory update.
Read how GPU Programming with Triton explains block-level reduction →
CUDA for LLMs
A block-level reduction combines values from threads within a GPU thread block into an aggregate result.
- How it is used
- Uses shared memory to extend warp-level reductions across a block in GEMV and parallel top-k kernels.
- How deep it goes
- Works through warp partials, shared-memory exchange, and synchronization, without detailing every helper implementation.
- Examples
- Uses a GEMV row sum and repeated top-k maximum-and-index reductions.
- What you can do after
- Apply it in CUDA kernels to combine thread-local values across a block.
Related concepts
- reduction treeused by · A reduction tree is a structure that combines values through successive stages of partial results to produce an aggregate.
- warp shuffleuses · A warp shuffle is a GPU operation that exchanges register-held values directly among lanes within the same warp.
- warp-level reductionhas part · A warp-level reduction combines values held by threads in a single warp into one aggregate result using warp-synchronous operations.
- reduction operationkind of · A reduction operation combines multiple input values according to a rule to produce fewer results, often a single aggregate.
- shared-memory reductionhas part · A shared-memory reduction combines GPU thread values through shared memory and synchronization between reduction steps.
- top-k kernelused by · The top-k kernel is the example CUDA implementation that extracts expert scores and indices for each token.
Questions about block-level reduction
What is block-level reduction?
A block-level reduction combines values across threads within a GPU thread block into an aggregate result.
Which Manning books explain block-level reduction?
GPU Programming with Triton and CUDA for LLMs.
What is block-level reduction a kind of?
Block-level reduction is a kind of reduction operation. A reduction operation combines multiple input values according to a rule to produce fewer results, often a single aggregate.
What are the parts of block-level reduction?
Block-level reduction includes warp-level reduction and shared-memory reduction.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

