liveBook · Manning

Chapter 3: Fused kernels and parallel patterns

What you will learn in chapter 3 of GPU Programming with Triton

The introduction recalls the independent per-program pattern of earlier Triton kernels and frames the move toward more complex workloads.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction recalls the independent per-program pattern of earlier Triton kernels and frames the move toward more complex workloads.

Key terms: program instances, slice of the input

3.1 The memory wall and kernel fusion

This section introduces the memory wall as a reason that cleanly composed, highly parallel machine-learning operations can underperform on a GPU.

Key terms: memory wall, on-chip storage, global memory, DRAM

3.1.1 Embarrassingly parallel tasks: the baseline GPU pattern

This section uses element-wise operations to explain embarrassingly parallel work and why it maps naturally to GPU execution.

Key terms: embarrassingly parallel, element-wise operations, warps, Single Instruction, Multiple Threads (SIMT)

3.1.2 SiLU as an embarrassingly parallel kernel

The section uses SiLU to show how an elementwise Triton kernel scales across independent blocks, then explains the memory cost of chaining separate kernels and clarifies floating-point performance terminology.

Key terms: SiLU, sigmoid, embarrassingly parallel, program instance, block, registers, global memory, grid, triton.cdiv, intermediate

3.1.3 Arithmetic intensity and the memory wall

Defines arithmetic intensity and explains how the growing gap between GPU compute throughput and memory bandwidth makes data movement a limit on kernel performance.

Key terms: arithmetic intensity, FLOPs, global memory, peak arithmetic intensity, memory bandwidth, memory bound, arithmetic bandwidth, memory wall

3.1.4 Compute-bound vs. memory-bound

This section explains how arithmetic intensity, relative to GPU hardware capability, determines whether a kernel is limited by memory bandwidth or computation.

Key terms: arithmetic intensity, memory-bound, compute-bound, memory bandwidth, HBM, ALUs

3.1.5 Roofline model

The section uses the roofline model to show both what limits a GPU kernel’s performance and how much room it has to improve.

Key terms: roofline model, arithmetic intensity, compute roof, memory roof, memory-bound, compute-bound, optimization headroom, intersection of the two roofs

3.2 The GPU Memory System

Introduces the GPU’s five-level memory hierarchy and explains why the location of data can determine whether a kernel computes or stalls.

Key terms: registers, shared memory, L1 cache, L2 cache, global memory, register file, HBM3 global memory, latency, bandwidth, intermediate results

3.2.1 Registers hold what matters right now

This section explains how fast, private registers support GPU computation and why their limited capacity creates performance trade-offs for Triton kernels.

Key terms: registers, Arithmetic Logic Units (ALUs), register file, register usage, fused kernels, intermediate values, register spilling, local memory, L1 and L2 caches, occupancy

3.2.2 Shared memory enables cooperation

Explains how shared memory lets threads in a block cooperate, reuse and rearrange data, and how Triton handles that storage.

Key terms: shared memory, thread block, registers, global memory, staging area, tiles, memory access patterns, L1 cache, unified data cache, explicit data placement

3.2.3 Caches provide transparent speedup

Explains how the GPU’s L1 and L2 caches automatically speed access to global memory, and why L2 bandwidth matters.

Key terms: caches, shared memory, global memory, DRAM, L1 cache, SM, register, L2, HBM, L2 cache

3.2.4 Global memory trades capacity for latency

Explains why global memory can offer high bandwidth yet have high access latency, and why keeping intermediate results in registers makes kernel fusion valuable.

Key terms: global memory, DRAM, SM, L2 cache, memory controller, latency, bandwidth, requests in flight, properly aligned, kernel fusion

3.3 Kernel fusion: the solution

This section explains how kernel fusion reduces global-memory traffic, quantifies the benefit for a SiLU-and-multiplication chain, and shows how the same idea applies to attention.

Key terms: kernel fusion, global memory, registers, intermediate results, arithmetic intensity, memory accesses, float32, memory-bound, kernel launch overhead, occupancy

3.3.1 Case Study: SwiGLU

This section uses SwiGLU to show how fusing elementwise operations avoids materializing an intermediate result and reduces global-memory traffic.

Key terms: SwiGLU, gate, value, up-projection, Swish, SiLU, materialization, global memory, arithmetic intensity, memory-bound

3.4 Control flow: masking over branching

Introduces why data-dependent conditional logic is needed in GPU kernels and why branching can be problematic for lockstep warps.

Key terms: straight-line execution path, conditional logic, if-else, warp, warp divergence

3.4.1 Warp divergence

Explains why different branches within a GPU warp execute serially and how that affects performance.

Key terms: warp, SIMT execution, Single Instruction, Multiple Threads, warp divergence, ReLU, branch, throughput

3.4.2 Predicated execution with tl.where

Explains how tl.where selects values per lane without divergent branching, and why candidate computation and masked memory access still matter.

Key terms: tl.where, boolean mask, per-lane select, warp, if statement, element-wise conditions, program ID, tile-level constant, warp divergence, candidate values

3.5 Advanced indexing: coordinate mapping (RoPE)

The section introduces coordinate mapping: turning an algorithm’s logical positions into addresses in flat GPU memory, especially when computation and data layout do not align neatly.

Key terms: GPU memory, indexing arithmetic, tl.load(ptr + offset), program ID, block size, coordinate mapping, RoPE, memory offsets, matrix transposition

3.5.1 Positional encoding in transformers

This section explains why transformers need positional information, how RoPE rotates paired query and key features to make attention sensitive to relative position, and what that implies for a GPU kernel.

Key terms: attention, positional encodings, absolute positions, relative distances, Rotary Positional Embeddings (RoPE), feature pairs, rotation angle, base frequency, multi-scale structure, dot product

3.5.2 The memory layout challenge

This section explains how RoPE’s split feature layout and shared rotation frequencies create two different row widths that a kernel must map correctly.

Key terms: split layout, Shared-Frequency design, cos and sin tensors, pid, sequence position m, input row, cached table row, strides, offset calculation

3.5.3 Building the kernel

This section builds a shared-frequency RoPE kernel by mapping sequence positions and feature pairs to the correct input and angle offsets.

Key terms: pid, half_dim, local_idx, BLOCK_SIZE, mask, coordinate mapping, row-major, row_start, first_offset, second_offset

3.5.4 Why coordinate mapping matters

Uses RoPE to show why GPU kernels need explicit coordinate-to-memory mapping and a way to check that mapping before coding.

Key terms: coordinate mapping, memory offsets, block-level offsets, program ID, program instance, stride, feature_dim, cos, sin

3.6 Exercises

3.6.1 Know thy GPU

3.6.2 Value clipping

3.6.3 Full elementwise RoPE

3.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 2: Write your first kernel in Triton Chapter 4: Reduction patterns →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.