3 Fused kernels and parallel patterns

In the previous chapter, you wrote your first Triton kernels and saw how a GPU launches thousands of program instances to process data in parallel. Those kernels followed a simple pattern: each program instance loaded its slice of the input, performed a computation, and stored the result. No instance needed to know what any other instance was doing. That simplicity made the mechanics of kernel launches, pointer arithmetic, and masking concrete, but it also left a question open: what happens when real workloads are more complex than a single operation applied independently to every element?

3.1 The memory wall and kernel fusion

Real machine learning programs almost never consist of a single operation. A typical forward pass applies a sequence of small steps such as element-wise transforms, scaling, masking, and normalization. In Python, these steps read cleanly: each line expresses a clear mathematical idea, and composing them feels natural. From the programmer’s point of view, the code appears simple and modular.

3.1.1 Embarrassingly parallel tasks: the baseline GPU pattern

3.1.2 SiLU as an embarrassingly parallel kernel

3.1.3 Arithmetic intensity and the memory wall

3.1.4 Compute-bound vs. memory-bound

3.1.5 Roofline model

3.2 The GPU Memory System

3.2.1 Registers hold what matters right now

3.2.2 Shared memory enables cooperation

3.2.3 Caches provide transparent speedup

3.2.4 Global memory trades capacity for latency

3.3 Kernel fusion: the solution

3.3.1 Case Study: SwiGLU

3.4 Control flow: masking over branching

3.4.1 Warp divergence

3.4.2 Predicated execution with tl.where

3.5 Advanced indexing: coordinate mapping (RoPE)

3.5.1 Positional encoding in transformers

3.5.2 The memory layout challenge

3.5.3 Building the kernel