liveBook · Manning

Chapter 4: Reduction patterns

What you will learn in chapter 4 of GPU Programming with Triton

Introduces reductions as GPU collective operations that turn distributed values into a shared statistic through coordinated communication.

After this chapter you will be able to

What it builds on

Section by section

Introduction

Introduces reductions as GPU collective operations that turn distributed values into a shared statistic through coordinated communication.

Key terms: reduction, collective operation, global statistics, parallel tree, partial results, warp, shared memory, global memory

4.1 The mechanics of reduction

This section explains why reductions require combining partial results, how a tree shortens that work, and how communication costs depend on GPU hardware hierarchy.

Key terms: reduction, combining operation, partial results, elementwise kernel, tree reduction, partial sums, critical path, lanes, warp, block

4.1.1 Floating-point associativity and numerical concerns

This section explains why floating-point reduction order affects results and introduces ways to reduce numerical error.

Key terms: associative, finite precision, rounding errors, tree reduction, tile shape, warps, bitwise identical, atomic floating-point updates, higher-precision accumulation, fp32

4.1.2 Triton’s primitives for reduction

This section explains the scope and API of Triton’s tile-local reductions, then shows how to build row-wise and grid-wide reductions with accumulation, multiple passes, or atomics.

Key terms: tl.tensor, program instance, tile, axis, tl.sum, keep_dims, broadcast, row-wise, column blocks, accumulators

4.1.3 Atomic operations

This section explains when Triton kernels need atomic writes, why contention makes them costly, and where Triton’s automatic reduction synchronization stops.

Key terms: atomic operations, tl.atomic_add, tl.store, data-dependent write destinations, L2 cache, contention, tl.sum, reduce first, then write atomically, pointer, val

4.1.4 RMS normalization in Triton

This section uses RMSNorm to show how Triton reduces a row to a scalar statistic, applies it to each element, and extends the kernel from one vector to a batch of rows.

Key terms: RMSNorm, inverse RMS, learned scale, reduce-then-map, ε, two passes, registers, shared memory, global memory, occupancy

4.2 Numerical stability and chained reductions (Softmax)

Introduces softmax as a probability normalization whose GPU implementation requires a vector-wide sum and attention to exponential overflow.

Key terms: softmax, logits, probability distribution, numerical stability, exponential, overflow, max reduction, denominator, sum, reduction

4.2.1 The "exploding exponential" problem

Explains why naive softmax exponentials can overflow and how subtracting the input maximum keeps the computation in range without changing the probabilities.

Key terms: float32, float16, overflow, dynamic range, softmax, invariance, maximum value, underflow

4.2.2 Case study: softmax in Triton

This section uses an unfused softmax baseline to show the cost of global reductions and intermediate memory traffic, then explains the locality needed to fuse the pipeline.

Key terms: unfused baseline, hierarchical reduction, partial value, reduction tree, global memory, intermediate tensor, memory bound, kernel launches, HBM, tl.max

4.2.3 Fused softmax kernel

This section explains when softmax reductions can be fused into one Triton program, how the kernel handles masked lanes, and why fusion improves performance.

Key terms: softmax axis, BLOCK_SIZE, program instance, tl.max, tl.sum, fusion, masking, other=-inf, tl.exp, max reduction

4.3 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 3: Fused kernels and parallel patterns Chapter 5: Higher-dimensional kernels →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.