liveBook · Manning

Chapter 5: Higher-dimensional kernels

What you will learn in chapter 5 of GPU Programming with Triton

The chapter introduction explains why matrix multiplication matters in deep learning and why matrix kernels require thinking about both logical dimensions and linear memory.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduction explains why matrix multiplication matters in deep learning and why matrix kernels require thinking about both logical dimensions and linear memory.

Key terms: layout, strides, tile shapes, program ordering, on-chip, matrix multiplication, linear layers, attention projections, convolution, specialized matrix hardware

5.1 Row-major and column-major ordering

This section explains how row-major and column-major layouts map the same matrix to different flat addresses and why traversal order affects memory-access performance.

Key terms: row-major order, column-major order, flat index, logical matrix, row-major matrix, warp, memory segment, memory transaction, Numba, @njit

5.2 Strides: the language of tensor layout

This section explains how strides map logical tensor indices onto flat memory and how changing that mapping can give the same buffer a different tensor interpretation.

Key terms: memory layout, rows, columns, logical index, stride, physical offset, base pointer, physical memory, layout rule, transpose

5.2.1 A concrete example

Uses a small matrix to show how strides turn logical coordinates into flat-memory offsets and why kernels need runtime strides as well as shape.

Key terms: row-major, stride tuple, flat storage index, offset, shape, runtime strides, contiguous row-major, transpose, stepped slice

5.2.2 The same memory, two tensors

This section shows how a transposed tensor can share its original storage by changing shape and strides, and when a copy is needed instead.

Key terms: transposed view, underlying storage, shape, strides, address formula, physical offset, row-major order, column-major order, .contiguous(), contiguous row-major storage

5.2.3 Stride interpretation and special values

This section explains how strides map tensor indices to storage addresses, how special stride values affect access, and how contiguous strides are determined.

Key terms: stride tuple, base pointer, runtime strides, physical offset, PyTorch view, shape, strides, underlying buffer, stride 1, warp

5.2.4 Memory coalescing: why access patterns affect performance

This section explains how a tensor’s stride affects the number of global-memory transactions needed to serve a warp.

Key terms: memory coalescing, warp, lane, sector, memory transaction, FP32, stride, useful-byte fraction, coalesced access, uncoalesced access

5.3 Matrix addition

Introduces matrix addition as an independent elementwise operation and explains what changes when mapping it from a 1D kernel to a 2D Triton kernel.

Key terms: matrix addition, elementwise, tiles, program IDs, row and column offsets, 2D pointer array, 2D mask

5.3.1 Building the first 2D kernel

Introduces how matrix addition moves from a one-dimensional vector-add pattern to two-dimensional tile coordinates, while noting when a flat-buffer approach also works.

Key terms: matrix tile, row coordinate, column coordinate, tensor strides, flat buffer, program_id(0), BLOCK_SIZE, 2D grid, ownership structure

5.3.2 2D program ID decomposition and tile indexing

This section shows how a two-dimensional Triton grid assigns matrix tiles to programs and how broadcasting row and column offsets creates pointers to every element in a tile.

Key terms: 2D launch grid, pid_m, pid_n, BLOCK_M, BLOCK_N, tile, row_offs, col_offs, tl.arange, tl.expand_dims

5.3.3 Boundary masking for non-tile-aligned shapes

Explains how a two-dimensional mask keeps edge tiles of a matrix-addition kernel from accessing positions outside the matrix.

Key terms: boundary masking, row_offs, col_offs, BLOCK_M, BLOCK_N, broadcasting, tl.load, tl.store, other=0.0, masked lane

5.3.4 The complete matrix-addition kernel

This subsection assembles the previously introduced pieces into a tiled matrix-addition kernel.

Key terms: row-major, C-contiguous, program instance, BLOCK_M, BLOCK_N, tile

5.4 Matrix transpose

Introduces matrix transpose as a coordinate swap whose scattered writes motivate moving the swap on chip.

Key terms: transpose, row-major, coordinates, global-memory address, tiled version, on chip, coalesced

5.4.1 Naive transpose kernel

This section builds a tiled Triton matrix transpose and explains why its straightforward layout produces contiguous reads but strided writes.

Key terms: BLOCK_M, BLOCK_N, pid_m, pid_n, rows, cols, transpose, tensor strides, x_ptrs, y_ptrs

5.4.2 Why uncoalesced writes hurt

Explains why the naive matrix transpose’s strided output stores require more memory transactions than contiguous stores.

Key terms: warp, FP32, stride_ym, 32-byte segments, coalesced, strided

5.4.3 Staging the transpose through shared memory

This section explains how on-chip reordering makes both sides of a tiled matrix transpose coalesced, then shows how Triton expresses the transpose with block pointers and a logical tile transpose.

Key terms: tiled transpose, shared memory, coalesced, global memory, barrier, load phase, read phase, shared-memory buffer, tl.make_block_ptr, x_block

5.4.4 Bank conflicts and the one-column padding fix

This section explains why column reads from a square shared-memory tile cause bank conflicts, how one-column padding avoids them, and why that fix should not be assumed to occur in the shown Triton kernel.

Key terms: shared-memory bank, 32-way bank conflict, broadcast, effective bandwidth, one column of padding, [32][33], bank, conflict-free, order=(1, 0), physical shared-memory array

5.5 Matrix multiplication

This section defines matrix multiplication’s dimensions and work, then explains why repeated data use and measured throughput matter for GPU performance.

Key terms: transformer block, attention, feed-forward network, matrix multiplication, throughput, M, N, K, contraction dimension, dot product

5.5.1 Extending matrix multiplication with scaling and accumulation

This section defines GEMM as scaled matrix multiplication with accumulation, shows how it supports gradient accumulation, and explains the β = 0 kernel path.

Key terms: GEMM, α, β, general matrix-matrix multiplication, weight-gradient matrix, gradient buffer, β = 0, runtime scalar, warp divergence, tl.constexpr meta-parameter

5.5.2 Naive matrix multiplication kernel

This section builds a correct one-output-per-program matrix multiplication kernel and measures how far its performance falls behind torch.matmul.

Key terms: program instance, launch grid, BLOCK_K, acc, stride, base pointer, row-major, FP32, FP16, BF16

5.5.3 Arithmetic intensity and the roofline view

The section calculates the naive FP32 matrix-multiplication kernel’s arithmetic intensity and uses a roofline estimate to explain why it is memory-bound.

Key terms: arithmetic intensity, FP32, FMA, FLOP/byte, program instances, global memory traffic, roofline model, ridge point, memory-bound, memory bandwidth

5.5.4 Reusing values across output elements

This section explains how tiled matrix multiplication reuses input values across output elements, how a program accumulates one output tile, and why tile size affects arithmetic intensity and resource use.

Key terms: output tile, reuse, arithmetic intensity, BLOCK_M, BLOCK_N, BLOCK_K, accumulator, tl.dot, square output tiles, rectangular tile

5.5.5 The first tiled kernel in Triton

This section walks through a first tiled Triton matrix-multiplication kernel that computes one output tile per program under exact-divisibility assumptions.

Key terms: C tile, pid_m, pid_n, offs_m, offs_n, K-strip, BLOCK_M, BLOCK_N, BLOCK_K, boundary masks

5.6 The Triton primitives behind the tiled kernel

Introduces the tl.dot operation that performs the main arithmetic in a tiled matrix multiplication kernel.

Key terms: tl.dot, 2D blocks, accumulator, tile-level programming model

5.6.1 The tl.dot block contraction

This subsection explains what tl.dot computes on matrix tiles and how a tiled kernel uses it while handling memory traversal and accumulation separately.

Key terms: tl.dot, contracted dimension, BLOCK_K, BLOCK_M, BLOCK_N, batch dimension, block matrix operation, accumulator, matrix multiply-accumulate, K slice

5.6.2 Describing tiles with tl.make_block_ptr

This section shows how tl.make_block_ptr describes matrix-multiplication tiles without manually constructing pointer arrays and masks.

Key terms: tl.make_block_ptr, block-pointer descriptor, shape, strides, starting offset, block shape, A, B, BLOCK_M, BLOCK_K

5.6.3 Advancing tiles with tl.advance

Explains how tl.advance moves block-pointer descriptors through the shared K dimension of an aligned tiled matrix multiplication, and how that traversal compares with manual pointers.

Key terms: tl.advance, descriptor, offset tuple, element offsets, shared K dimension, BLOCK_K, a_block_ptr, b_block_ptr, tl.dot, manual-pointer kernel

5.7 Why tiled matrix multiplication uses Tensor Cores

This section explains why a tiled Triton matrix multiplication can use Tensor Cores, how `tl.dot` relates to native matrix instructions, and how those instructions fit into the rest of a kernel.

Key terms: CUDA cores, Tensor Cores, FMA, MMA, warp scheduler, program tile, native matrix shape, fragments, tl.dot, target backend

5.8 From aligned tiles to an optimized matrix multiplication

This section identifies why the earlier tiled matrix-multiplication kernels are not yet general before introducing the improvements to come.

Key terms: tiled kernels, tile boundary, tile order, launch configuration

5.8.1 Handling incomplete edge tiles

Explains how masks and zero-filled input loads let a tiled matrix multiplication handle incomplete edge tiles safely.

Key terms: K, BLOCK_K, mask, tl.dot, accumulator, M, N, load masks, store mask, other=0.0

5.8.2 Reordering output tiles to reuse L2 data

This subsection explains how grouping output tiles by program ID can improve matrix multiplication’s L2 cache reuse without changing its result.

Key terms: row-major mapping, grouped mapping, GROUP_M, L2 cache, HBM, A tile, program IDs, matrix shape, tile sizes

5.8.3 Letting the autotuner choose a launch configuration

This section explains how Triton selects a matrix-multiplication launch configuration for each shape and why benchmarks must account for tuning time.

Key terms: launch configuration, N tile, autotuner, triton.Config, BLOCK_M, BLOCK_N, BLOCK_K, num_warps, num_stages, first launch

5.8.4 Adding compiler hints carefully

Explains what compiler hints tell the matrix-multiplication compiler, why their promises must be true, and why any performance benefit must be measured.

Key terms: compiler hints, tl.assume, tl.multiple_of, tl.max_contiguous, M and N offsets, promises, runtime checks, host wrapper, positive strides, benchmark

5.8.5 Final optimized matrix multiplication kernel

Summarizes how the final optimized matrix multiplication kernel assigns, computes, and protects output tiles, and how its host wrapper validates inputs and launches it.

Key terms: launch configuration, grouped mapping, masks, K loop, tl.dot, wrapper, contiguous GPU matrices, FP16, BF16, inner dimensions

5.8.6 Checking correctness across shapes and dtypes

This section explains how to check the optimized matrix multiplication kernel against an independent reference across shapes and dtypes while allowing for floating-point differences.

Key terms: PyTorch calculation, reference, indexing mistake, M, N, K, remainders, FP16, BF16, FP32

5.9 Benchmarking the matrix multiplication progression

This section benchmarks the matrix-multiplication kernels and uses the results to compare their performance across shapes and on a roofline plot.

Key terms: warm-up, medians, 2MNK operation count, scalar tl.sum kernel, tiling, block-pointer version, grouped ordering, autotuning, torch.matmul, FP16

5.10 Summary

Concepts explained in this chapter

← Chapter 4: Reduction patterns

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.