Chapter 5: Higher-dimensional kernels
The chapter introduction explains why matrix multiplication matters in deep learning and why matrix kernels require thinking about both logical dimensions and linear memory.
After this chapter you will be able to
- Matrix Layout and Memory. Explain why a matrix kernel must account for both the matrix’s two-dimensional shape and its linear representation in memory, and how layout, strides, tile shapes, and program ordering affect loading and on-chip reuse.
- Matrix layout and flat indices. Explain how row-major and column-major order place the same logical matrix in memory, and compute a flat index for each layout.
- Traversal and memory efficiency. Explain why walking across rows of a row-major matrix generally accesses memory more efficiently than walking down its columns, including the GPU access pattern described here.
- Stride-based address calculation. Describe how logical indices and strides determine an element's flat offset, and explain what stride values of 1, 4, and 0 mean.
- Multiple views of one buffer. Explain how different layout rules can interpret the same physical buffer as different tensors without moving its bytes.
- Stride-based address calculation. Explain how the row-major matrix’s strides (4, 1) locate an element in flat storage, including why position (2, 1) has offset 9.
- Runtime strides and tensor shape. Distinguish the information supplied by shape from that supplied by strides, and explain why a kernel should receive runtime strides for contiguous, transposed, or sliced tensors.
- Transpose as a tensor view. Explain that calling .t() changes the tensor’s logical shape and swaps its strides without moving or copying its underlying data.
- Strides and shared offsets. Trace how the original and transposed coordinates reach the same physical element, and describe how passing different strides changes a kernel’s interpretation of shared storage.
- Stride-based element addresses. Explain how a base pointer and runtime strides locate a 2D tensor element, and why a Triton kernel should use those strides rather than infer layout from shape.
- Zero-stride broadcasting. Explain why moving along a stride-0 dimension leaves the address unchanged, using the broadcast of an (N,) vector across an (M, N) tensor.
- Warp-wide memory coalescing. Explain how the GPU combines the addresses supplied by lanes in a warp for one load instruction and determines the number of memory transactions.
- Contiguous versus strided loads. Contrast the transactions and useful bytes for the section’s contiguous and stride-32 FP32 warp loads.
- Two-dimensional tile mapping. Describe the changes needed to map matrix addition onto a 2D output space: divide it into tiles, use different program IDs for row and column offsets, combine those offsets into a 2D pointer array, and mask dimensions that do not divide evenly by the tile size.
- Two-dimensional matrix tile coordinates. Explain why a matrix tile uses row and column coordinates, and how a program turns those coordinates into addresses using tensor strides.
- Two-dimensional tile ownership. Explain how a two-dimensional launch grid uses pid_m and pid_n to assign each program one output tile, and how those IDs determine the tile’s starting row and column.
- Broadcast tile pointer construction. Describe how absolute row and column offset vectors are expanded, scaled by their strides, and broadcast to form a pointer for every element of a matrix tile.
- Two-dimensional boundary mask. Explain why non-tile-aligned matrices need boundary masking and how row and column validity conditions combine to identify valid positions in a tile.
- Complete tiled matrix addition. Trace how the example adds two row-major matrices: each program instance selects a BLOCK_M by BLOCK_N tile, forms offsets, a mask, and pointers, then loads, adds, and stores values.
- Transpose coordinate mapping. Explain how transpose differs from matrix addition: values read at (row, column) are written with the coordinates reversed.
- Transpose write bottleneck. Explain why the stores, rather than the number of loads and stores, make the straightforward transpose expensive, and state the tiled version's high-level remedy.
- Tiled transpose coordinate grid. Describe how a program instance selects a tile, forms row and column coordinates, and masks positions beyond the matrix edge. Explain how broadcasting the row and column vectors creates coordinates for every position in the tile.
- Stride based transpose mapping. Explain how the kernel copies an M by N input into an N by M output by exchanging destination coordinates, using the supplied tensor strides in its input and output pointer expressions.
- Naive transpose access pattern. Explain why the kernel is called naive despite parallel tiled execution: neighboring lanes read contiguous input values but write to output addresses a full output row apart.
- Strided transpose store cost. Explain how the naive transpose’s output stride makes a warp’s stores less coalesced, comparing the transactions needed for contiguous and strided FP32 writes. Use the 4096 × 4096 output example to illustrate the spacing between neighboring lane addresses.
- Coalesced transpose through on-chip staging. Explain why the naive transpose has poorly spaced output writes and how staging a tile on chip allows coalesced global-memory reads and writes.
- Logical tile transpose in Triton. Explain what tl.trans does to the loaded tile and how that differs from swapping row and column indices only in naive output pointers.
- Square tile bank conflicts. Explain why reading a column of a 32 × 32 shared-memory tile causes a 32-way bank conflict and how that affects transpose bandwidth.
- One-column padding fix. Explain how changing a shared-memory tile from 32 × 32 to 32 × 33 makes a column read conflict-free.
- Product dimensions and contraction. Describe the shapes of A, B, and C and how one output element is formed; explain why the shared K dimension must match.
- Repeated reads and tile reuse. Explain what a scalar-output implementation repeatedly reads and how allowing a loaded tile to serve multiple outputs changes that traffic.
- Work count and achieved throughput. State the section’s floating-point work count for matrix multiplication and apply it with elapsed time to calculate achieved TFLOP/s.
- Scaled matrix product accumulation. Explain the GEMM update C ← αAB + βC, including what each scalar controls and how ordinary multiplication and accumulation follow from particular values.
- Zero-beta kernel specialization. Explain why a GEMM kernel can omit the old-C load when β = 0, and distinguish a uniform runtime branch from a guaranteed specialized path.
- One output per program. Trace how the launch grid assigns each output element to a Triton program and how that program computes its dot product.
- Costs of independent outputs. Explain the access-pattern and data-reuse limitations of assigning only one output to each program.
- Per-output arithmetic intensity. Calculate the naive kernel’s arithmetic intensity from the work and memory traffic for one output element. Explain why it approaches 0.25 FLOP/byte for large K.
- Memory-bound roofline position. Use the stated H200 FP32 and bandwidth limits to explain why the naive kernel is memory-bound and estimate its bandwidth-limited throughput.
- Input reuse across outputs. Explain why assigning several output elements to one program avoids repeated input loads: outputs in the same row reuse values from A, while outputs in the same column reuse values from B.
- Output tile accumulation. Trace how one program computes a BLOCK_M × BLOCK_N tile of C by initializing an accumulator, processing K in BLOCK_K slices with A and B tiles, and writing the completed tile.
- Output tile computation. Trace how one program selects a tile of C, builds pointer tiles for A and B, accumulates products across K slices, and stores the completed result. Explain why the accumulator uses FP32.
- Tile-level dot product accumulation. Explain that acc = tl.dot(a, b, acc) multiplies two 2D blocks and adds their product to an accumulator, connecting Triton’s tile-level code to GPU matrix hardware.
- Block contraction shape contract. Explain which dimensions tl.dot contracts and what shape it returns for 2D and batched 3D inputs. Apply the matching-dimension rule to identify a valid or invalid pair of input tiles.
- Tile operation and accumulator. Describe tl.dot as a block matrix multiply-accumulate and distinguish its high-level tile operation from a sequence of scalar multiply-adds.
- K loop memory boundary. Trace how successive K slices contribute to an output tile and distinguish tl.dot's computation from the surrounding kernel's memory operations.
- Structured block-pointer descriptors. Explain how tl.make_block_ptr replaces manual pointer-array and mask setup with a descriptor for an N-dimensional memory block. Describe what the descriptor records and distinguish creating it from loading data.
- Block pointer advancement semantics. Explain what tl.advance changes in a block-pointer descriptor, how its offset tuple is interpreted, and why the returned descriptor is reassigned in a loop.
- Matching K tile movement. Trace how the A and B block pointers advance after each tl.dot so the next operation receives matching slices of their shared K dimension.
- Scalar versus matrix arithmetic. Explain why Tensor Cores are suited to tiled matrix multiplication rather than expressing the same work as scalar multiplies, adds, and reductions. Describe MMA as D = AB + C and account for its multiplication and accumulation work.
- Cooperative native matrix operations. Distinguish a Triton program tile from a native matrix instruction, and describe how threads collectively supply matrix operands and receive results.
- Conditional Tensor Core compilation. Explain what `tl.dot` tells Triton's compiler and why it does not guarantee a particular Tensor Core instruction.
- Shared kernel execution resources. Trace how one tiled K-loop iteration uses different SM resources, and explain why Tensor Cores can still wait for operands.
- Limits of earlier tiled kernels. Explain why the earlier tiled matrix-multiplication kernels are not yet general: they require dimensions to end at tile boundaries and use the same tile order and launch configuration for every shape.
- Incomplete K tile handling. Explain how the kernel handles a final K tile that extends beyond the matrices while still passing a complete tile to tl.dot.
- Edge load and store masks. Distinguish how load and store masks handle incomplete M and N tiles, and explain why masked input loads need other=0.0.
- Grouped output tile ordering. Explain how row-major and grouped program-ID mappings visit output tiles, and why grouped ordering can improve input-tile reuse.
- Shape-dependent launch choices. Explain why one launch configuration is not best for every matrix and why Triton measures several candidates instead of relying on one hand-picked choice.
- Autotune candidates and keys. Describe what a triton.Config candidate contains and how triton.autotune selects a configuration when M, N, or K changes.
- Compiler hint meanings. Explain that compiler hints state facts about the existing matrix multiplication rather than changing its computation. Describe the facts supplied by tl.assume, tl.multiple_of, and tl.max_contiguous and how the compiler may use them.
- Hint validity and benchmarking. Justify why the hints are valid for this kernel and why their performance effect still needs benchmarking.
- Final kernel job sequence. Explain how the decorator, grouped mapping, masks, and K loop each contribute to the final matrix multiplication kernel. State why the six launch configurations are a starting set rather than universal choices.
- Independent correctness reference. Explain why the final kernel is compared with a separate PyTorch calculation rather than only with an earlier custom kernel.
- Shape and dtype coverage. Describe the shape and dtype cases used to challenge the final kernel's correctness.
- Floating point comparison tolerance. Explain why the Triton and PyTorch results are compared with dtype-specific tolerance rather than bit-for-bit equality.
- Aligned kernel performance progression. Explain the measured progression from scalar matrix multiplication through tiled and optimized kernels on the aligned 4096 case.
- Conservative roofline placement. Explain why the optimized kernel’s roofline marker begins at a lower bound for operational intensity rather than at a measured intensity.
What it builds on
- The GPU execution model (chapter 1) · Introduces warp-based GPU execution.
- A complete Triton kernel: vector addition end to end (chapter 2) · The vector-addition kernel used as the 1D comparison.
- Getting a unique program ID (chapter 2) · Introduces program IDs in a 1D kernel.
- Handling edges with a mask (chapter 2) · Introduces masking at the edge of a 1D kernel.
- Computing the element offsets (chapter 2) · Offset vectors locate elements processed by a program.
- Arithmetic intensity and the memory wall (chapter 3) · The effect of memory traffic on GPU computation.
- Warp divergence (chapter 3) · The meaning of warp divergence when lanes follow different branches.
- Roofline model (chapter 3) · The roofline model.
Section by section
Introduction
The chapter introduction explains why matrix multiplication matters in deep learning and why matrix kernels require thinking about both logical dimensions and linear memory.
- Matrix Layout and Memory: Explain why a matrix kernel must account for both the matrix’s two-dimensional shape and its linear representation in memory, and how layout, strides, tile shapes, and program ordering affect loading and on-chip reuse.
Why it matters: This distinction is the starting point for reasoning about higher-dimensional kernels. - Matrix Multiplication in Deep Learning: Describe why matrix multiplication is a useful focus for this chapter, using the deep-learning operations named in the introduction.
Why it matters: It explains why the chapter uses matrix multiplication to connect indexing concepts with GPU hardware.
Key terms: layout, strides, tile shapes, program ordering, on-chip, matrix multiplication, linear layers, attention projections, convolution, specialized matrix hardware
5.1 Row-major and column-major ordering
This section explains how row-major and column-major layouts map the same matrix to different flat addresses and why traversal order affects memory-access performance.
- Matrix layout and flat indices: Explain how row-major and column-major order place the same logical matrix in memory, and compute a flat index for each layout.
Why it matters: A kernel must turn a row and column into the correct location in its data buffer. - Traversal and memory efficiency: Explain why walking across rows of a row-major matrix generally accesses memory more efficiently than walking down its columns, including the GPU access pattern described here.
Why it matters: An indexing pattern can be correct yet make a kernel use memory bandwidth poorly. - CPU traversal benchmark: Describe what the section's two summation loops compare, what the reported timings show, and why the example is a CPU cache demonstration rather than a GPU benchmark.
Why it matters: The benchmark makes the cost of traversal order concrete without mistaking CPU cache results for GPU measurements.
Key terms: row-major order, column-major order, flat index, logical matrix, row-major matrix, warp, memory segment, memory transaction, Numba, @njit
5.2 Strides: the language of tensor layout
This section explains how strides map logical tensor indices onto flat memory and how changing that mapping can give the same buffer a different tensor interpretation.
- Logical structure in flat memory: Explain why rows and columns are properties of a memory layout rather than structures recognized by hardware.
Why it matters: It explains why a tensor needs a layout rule in addition to its stored values. - Stride-based address calculation: Describe how logical indices and strides determine an element's flat offset, and explain what stride values of 1, 4, and 0 mean.
Why it matters: This calculation connects a tensor coordinate to the memory location the GPU fetches. - Multiple views of one buffer: Explain how different layout rules can interpret the same physical buffer as different tensors without moving its bytes.
Why it matters: It shows why changing a tensor's interpretation need not require copying its values.
Key terms: memory layout, rows, columns, logical index, stride, physical offset, base pointer, physical memory, layout rule, transpose
5.2.1 A concrete example
Uses a small matrix to show how strides turn logical coordinates into flat-memory offsets and why kernels need runtime strides as well as shape.
- Stride-based address calculation: Explain how the row-major matrix’s strides (4, 1) locate an element in flat storage, including why position (2, 1) has offset 9.
Why it matters: This connects a matrix position to the location a kernel must access in memory. - Runtime strides and tensor shape: Distinguish the information supplied by shape from that supplied by strides, and explain why a kernel should receive runtime strides for contiguous, transposed, or sliced tensors.
Why it matters: Using the received tensor’s strides prevents a kernel from addressing the wrong elements when the layout changes.
Key terms: row-major, stride tuple, flat storage index, offset, shape, runtime strides, contiguous row-major, transpose, stepped slice
5.2.2 The same memory, two tensors
This section shows how a transposed tensor can share its original storage by changing shape and strides, and when a copy is needed instead.
- Transpose as a tensor view: Explain that calling .t() changes the tensor’s logical shape and swaps its strides without moving or copying its underlying data.
Why it matters: A transpose does not necessarily incur the cost of copying tensor data. - Strides and shared offsets: Trace how the original and transposed coordinates reach the same physical element, and describe how passing different strides changes a kernel’s interpretation of shared storage.
Why it matters: Stride-based addressing lets a kernel use either view without assuming the data was rearranged. - When contiguous storage requires copying: Distinguish when code can retain the transposed view from when it must make a contiguous copy.
Why it matters: This distinction helps avoid an unnecessary copy while recognizing when an operation needs one.
Key terms: transposed view, underlying storage, shape, strides, address formula, physical offset, row-major order, column-major order, .contiguous(), contiguous row-major storage
5.2.3 Stride interpretation and special values
This section explains how strides map tensor indices to storage addresses, how special stride values affect access, and how contiguous strides are determined.
- Stride-based element addresses: Explain how a base pointer and runtime strides locate a 2D tensor element, and why a Triton kernel should use those strides rather than infer layout from shape.
Why it matters: Runtime strides let one kernel address tensors with different layouts correctly. - Views and shared storage: Describe what shape, base pointer, and strides each specify, and how a PyTorch view can change logical indexing without copying its buffer.
Why it matters: A tensor’s logical appearance does not by itself reveal how it walks memory. - Unit-stride access: Explain what stride 1 means for neighboring elements and why accessing along that dimension can help a warp coalesce loads.
Why it matters: Finding the stride-1 axis connects tensor layout to efficient memory access. - Zero-stride broadcasting: Explain why moving along a stride-0 dimension leaves the address unchanged, using the broadcast of an (N,) vector across an (M, N) tensor.
Why it matters: Zero strides explain how broadcasting presents repeated values without storing repeated copies. - Canonical contiguous strides: State how to derive row-major contiguous strides from a shape and explain what the section says PyTorch’s .is_contiguous() checks.
Why it matters: The canonical pattern provides a reference for recognizing when valid view strides are not contiguous row-major strides.
Key terms: stride tuple, base pointer, runtime strides, physical offset, PyTorch view, shape, strides, underlying buffer, stride 1, warp
5.2.4 Memory coalescing: why access patterns affect performance
This section explains how a tensor’s stride affects the number of global-memory transactions needed to serve a warp.
- Warp-wide memory coalescing: Explain how the GPU combines the addresses supplied by lanes in a warp for one load instruction and determines the number of memory transactions.
Why it matters: Counting occupied sectors lets you reason about memory traffic from an access pattern. - Contiguous versus strided loads: Contrast the transactions and useful bytes for the section’s contiguous and stride-32 FP32 warp loads.
Why it matters: The examples show why a strided load can move much more data than the computation uses. - Layout-driven access cost: Explain how the stride followed across a warp affects coalescing and why reading a transposed view can change the access cost.
Why it matters: A view can change memory-access efficiency even when the values being read are the same.
Key terms: memory coalescing, warp, lane, sector, memory transaction, FP32, stride, useful-byte fraction, coalesced access, uncoalesced access
5.3 Matrix addition
Introduces matrix addition as an independent elementwise operation and explains what changes when mapping it from a 1D kernel to a 2D Triton kernel.
- Independent matrix addition: Explain that matrix addition computes C[i, j] = A[i, j] + B[i, j] for matrices of shape (M, N), and that output positions can be computed independently.
Why it matters: This independence makes matrix addition a straightforward first example of a 2D kernel. - Two-dimensional tile mapping: Describe the changes needed to map matrix addition onto a 2D output space: divide it into tiles, use different program IDs for row and column offsets, combine those offsets into a 2D pointer array, and mask dimensions that do not divide evenly by the tile size.
Why it matters: This is the basic mapping used to organize work over a matrix, including its edges.
Key terms: matrix addition, elementwise, tiles, program IDs, row and column offsets, 2D pointer array, 2D mask
5.3.1 Building the first 2D kernel
Introduces how matrix addition moves from a one-dimensional vector-add pattern to two-dimensional tile coordinates, while noting when a flat-buffer approach also works.
- Two-dimensional matrix tile coordinates: Explain why a matrix tile uses row and column coordinates, and how a program turns those coordinates into addresses using tensor strides.
Why it matters: Row and column coordinates establish how a program identifies the matrix elements in its tile. - Flat addition versus 2D ownership: Distinguish the flat-buffer alternative for matrix addition from the chapter’s choice of a 2D grid, and explain why the chapter retains that grid.
Why it matters: The 2D grid is a deliberate teaching choice, not a claim that matrix addition cannot be flattened.
Key terms: matrix tile, row coordinate, column coordinate, tensor strides, flat buffer, program_id(0), BLOCK_SIZE, 2D grid, ownership structure
5.3.2 2D program ID decomposition and tile indexing
This section shows how a two-dimensional Triton grid assigns matrix tiles to programs and how broadcasting row and column offsets creates pointers to every element in a tile.
- Two-dimensional tile ownership: Explain how a two-dimensional launch grid uses pid_m and pid_n to assign each program one output tile, and how those IDs determine the tile’s starting row and column.
Why it matters: The program IDs tell each instance which part of the matrix it must compute. - Broadcast tile pointer construction: Describe how absolute row and column offset vectors are expanded, scaled by their strides, and broadcast to form a pointer for every element of a matrix tile.
Why it matters: Broadcasting turns two short coordinate vectors into addresses for a whole tile without listing each element separately.
Key terms: 2D launch grid, pid_m, pid_n, BLOCK_M, BLOCK_N, tile, row_offs, col_offs, tl.arange, tl.expand_dims
5.3.3 Boundary masking for non-tile-aligned shapes
Explains how a two-dimensional mask keeps edge tiles of a matrix-addition kernel from accessing positions outside the matrix.
- Two-dimensional boundary mask: Explain why non-tile-aligned matrices need boundary masking and how row and column validity conditions combine to identify valid positions in a tile.
Why it matters: This pattern lets tiled kernels handle matrix dimensions that do not divide evenly by the tile size. - Masked loads and stores: Describe what the boundary mask does in tl.load and tl.store, and justify using other=0.0 for this matrix addition.
Why it matters: Applying the mask to both memory operations prevents invalid reads and writes while preserving the valid addition results.
Key terms: boundary masking, row_offs, col_offs, BLOCK_M, BLOCK_N, broadcasting, tl.load, tl.store, other=0.0, masked lane
5.3.4 The complete matrix-addition kernel
This subsection assembles the previously introduced pieces into a tiled matrix-addition kernel.
- Complete tiled matrix addition: Trace how the example adds two row-major matrices: each program instance selects a BLOCK_M by BLOCK_N tile, forms offsets, a mask, and pointers, then loads, adds, and stores values.
Why it matters: It connects tile indexing and boundary masking to the complete matrix-addition operation.
Key terms: row-major, C-contiguous, program instance, BLOCK_M, BLOCK_N, tile
5.4 Matrix transpose
Introduces matrix transpose as a coordinate swap whose scattered writes motivate moving the swap on chip.
- Transpose coordinate mapping: Explain how transpose differs from matrix addition: values read at (row, column) are written with the coordinates reversed.
Why it matters: The coordinate swap explains why a simple transpose can have an unfavorable memory-access pattern. - Transpose write bottleneck: Explain why the stores, rather than the number of loads and stores, make the straightforward transpose expensive, and state the tiled version's high-level remedy.
Why it matters: This identifies the memory-access problem the tiled transpose is intended to solve.
Key terms: transpose, row-major, coordinates, global-memory address, tiled version, on chip, coalesced
5.4.1 Naive transpose kernel
This section builds a tiled Triton matrix transpose and explains why its straightforward layout produces contiguous reads but strided writes.
- Tiled transpose coordinate grid: Describe how a program instance selects a tile, forms row and column coordinates, and masks positions beyond the matrix edge. Explain how broadcasting the row and column vectors creates coordinates for every position in the tile.
Why it matters: The coordinate grid is how one Triton program addresses a whole transpose tile without nested loops. - Stride based transpose mapping: Explain how the kernel copies an M by N input into an N by M output by exchanging destination coordinates, using the supplied tensor strides in its input and output pointer expressions.
Why it matters: The pointer expressions show precisely where each input value goes in the transposed output. - Naive transpose access pattern: Explain why the kernel is called naive despite parallel tiled execution: neighboring lanes read contiguous input values but write to output addresses a full output row apart.
Why it matters: Correct coordinate mapping alone does not ensure efficient GPU memory access.
Key terms: BLOCK_M, BLOCK_N, pid_m, pid_n, rows, cols, transpose, tensor strides, x_ptrs, y_ptrs
5.4.2 Why uncoalesced writes hurt
Explains why the naive matrix transpose’s strided output stores require more memory transactions than contiguous stores.
- Strided transpose store cost: Explain how the naive transpose’s output stride makes a warp’s stores less coalesced, comparing the transactions needed for contiguous and strided FP32 writes. Use the 4096 × 4096 output example to illustrate the spacing between neighboring lane addresses.
Why it matters: It explains the performance problem that the subsequent transpose implementation addresses.
Key terms: warp, FP32, stride_ym, 32-byte segments, coalesced, strided
5.4.3 Staging the transpose through shared memory
This section explains how on-chip reordering makes both sides of a tiled matrix transpose coalesced, then shows how Triton expresses the transpose with block pointers and a logical tile transpose.
- Coalesced transpose through on-chip staging: Explain why the naive transpose has poorly spaced output writes and how staging a tile on chip allows coalesced global-memory reads and writes.
Why it matters: The key improvement is changing where the awkward traversal happens, not merely changing the tile size. - Shared-memory barrier between phases: Explain why threads must synchronize after filling the shared-memory tile and before reading it in transposed order.
Why it matters: The barrier makes the two-phase shared-memory reordering safe when threads exchange which values they use. - Input and output block pointers: Describe how the Triton example selects corresponding input and transposed output tiles with block pointers, and distinguish pointer creation from memory access.
Why it matters: Block pointers describe which rectangular regions participate in the transpose without themselves moving data. - Logical tile transpose in Triton: Explain what tl.trans does to the loaded tile and how that differs from swapping row and column indices only in naive output pointers.
Why it matters: The Triton code expresses the desired tile permutation without committing to the pseudocode's particular on-chip implementation.
Key terms: tiled transpose, shared memory, coalesced, global memory, barrier, load phase, read phase, shared-memory buffer, tl.make_block_ptr, x_block
5.4.4 Bank conflicts and the one-column padding fix
This section explains why column reads from a square shared-memory tile cause bank conflicts, how one-column padding avoids them, and why that fix should not be assumed to occur in the shown Triton kernel.
- Square tile bank conflicts: Explain why reading a column of a 32 × 32 shared-memory tile causes a 32-way bank conflict and how that affects transpose bandwidth.
Why it matters: It explains why staging a transpose through shared memory can still have a costly access pattern. - One-column padding fix: Explain how changing a shared-memory tile from 32 × 32 to 32 × 33 makes a column read conflict-free.
Why it matters: The padding changes where column elements land in shared-memory banks without changing the values being transposed. - Triton layout limitation: Distinguish the low-level padding example from what the Triton transpose kernel actually specifies.
Why it matters: It keeps the bank-padding explanation from becoming an unsupported claim about the Triton kernel's physical layout.
Key terms: shared-memory bank, 32-way bank conflict, broadcast, effective bandwidth, one column of padding, [32][33], bank, conflict-free, order=(1, 0), physical shared-memory array
5.5 Matrix multiplication
This section defines matrix multiplication’s dimensions and work, then explains why repeated data use and measured throughput matter for GPU performance.
- Matrix multiplication in transformers: Explain why matrix multiplication performance matters to transformer throughput, citing its use in attention, feed-forward networks, and training.
Why it matters: It connects the matrix operation to the performance of a whole transformer. - Product dimensions and contraction: Describe the shapes of A, B, and C and how one output element is formed; explain why the shared K dimension must match.
Why it matters: These dimension roles determine which values contribute to each output position. - Repeated reads and tile reuse: Explain what a scalar-output implementation repeatedly reads and how allowing a loaded tile to serve multiple outputs changes that traffic.
Why it matters: It identifies the redundant input traffic that tiling aims to reduce. - Work count and achieved throughput: State the section’s floating-point work count for matrix multiplication and apply it with elapsed time to calculate achieved TFLOP/s.
Why it matters: It gives a consistent way to compare measured matrix multiplication performance.
Key terms: transformer block, attention, feed-forward network, matrix multiplication, throughput, M, N, K, contraction dimension, dot product
5.5.1 Extending matrix multiplication with scaling and accumulation
This section defines GEMM as scaled matrix multiplication with accumulation, shows how it supports gradient accumulation, and explains the β = 0 kernel path.
- Scaled matrix product accumulation: Explain the GEMM update C ← αAB + βC, including what each scalar controls and how ordinary multiplication and accumulation follow from particular values.
Why it matters: The scalar choices determine whether a matrix product replaces C or builds on its existing values. - Gradient buffer accumulation: Describe how β controls whether a linear layer's weight-gradient contribution starts a new accumulation or adds to an existing gradient buffer.
Why it matters: This is a concrete reason to keep or discard C's previous values. - Zero-beta kernel specialization: Explain why a GEMM kernel can omit the old-C load when β = 0, and distinguish a uniform runtime branch from a guaranteed specialized path.
Why it matters: It separates avoiding warp divergence from guaranteeing a kernel that omits unnecessary work.
Key terms: GEMM, α, β, general matrix-matrix multiplication, weight-gradient matrix, gradient buffer, β = 0, runtime scalar, warp divergence, tl.constexpr meta-parameter
5.5.2 Naive matrix multiplication kernel
This section builds a correct one-output-per-program matrix multiplication kernel and measures how far its performance falls behind torch.matmul.
- One output per program: Trace how the launch grid assigns each output element to a Triton program and how that program computes its dot product.
Why it matters: This ownership rule makes the kernel easy to check, but also determines how much work and data loading each program performs. - Stride-based input addressing: Describe how a program locates its row of A and column of B, and why the host passes strides to the kernel.
Why it matters: Stride-based addressing keeps the baseline kernel usable with more than one input layout. - FP32 dot-product accumulation: Explain when the kernel converts input values to FP32 and why that ordering matters for FP16 or BF16 inputs.
Why it matters: The order of conversion makes this a useful scalar baseline for lower-precision inputs. - Costs of independent outputs: Explain the access-pattern and data-reuse limitations of assigning only one output to each program.
Why it matters: The main limitation is not correctness but loading data separately when it could contribute to multiple outputs. - Validated performance baseline: Describe why the naive kernel is established before optimization and what the comparison with torch.matmul shows.
Why it matters: Checking correctness and measuring against a practical reference gives optimization a trustworthy starting point.
Key terms: program instance, launch grid, BLOCK_K, acc, stride, base pointer, row-major, FP32, FP16, BF16
5.5.3 Arithmetic intensity and the roofline view
The section calculates the naive FP32 matrix-multiplication kernel’s arithmetic intensity and uses a roofline estimate to explain why it is memory-bound.
- Per-output arithmetic intensity: Calculate the naive kernel’s arithmetic intensity from the work and memory traffic for one output element. Explain why it approaches 0.25 FLOP/byte for large K.
Why it matters: This calculation puts the kernel’s memory use and computation on the same scale. - Repeated reads across outputs: Explain why the per-output intensity estimate also describes the whole naive kernel, despite changes in matrix dimensions.
Why it matters: It explains why making the matrices larger does not, by itself, improve this kernel’s calculated arithmetic intensity. - Memory-bound roofline position: Use the stated H200 FP32 and bandwidth limits to explain why the naive kernel is memory-bound and estimate its bandwidth-limited throughput.
Why it matters: The roofline comparison identifies memory bandwidth, rather than peak FP32 compute, as the naive kernel’s calculated performance limit.
Key terms: arithmetic intensity, FP32, FMA, FLOP/byte, program instances, global memory traffic, roofline model, ridge point, memory-bound, memory bandwidth
5.5.4 Reusing values across output elements
This section explains how tiled matrix multiplication reuses input values across output elements, how a program accumulates one output tile, and why tile size affects arithmetic intensity and resource use.
- Input reuse across outputs: Explain why assigning several output elements to one program avoids repeated input loads: outputs in the same row reuse values from A, while outputs in the same column reuse values from B.
Why it matters: This reuse is the reason a tiled program can make better use of memory bandwidth than the naive one-output-per-program approach. - Output tile accumulation: Trace how one program computes a BLOCK_M × BLOCK_N tile of C by initializing an accumulator, processing K in BLOCK_K slices with A and B tiles, and writing the completed tile.
Why it matters: It connects input reuse to the actual work assigned to each program. - Square tile reuse balance: Justify why the first tiled kernel uses equal BLOCK_M and BLOCK_N dimensions, while noting that rectangular tiles can still run faster for a particular shape or GPU.
Why it matters: Square tiles give a clear starting point for reuse, not a guarantee of the fastest configuration. - Tile intensity and resource tradeoff: Explain the section's large-K arithmetic-intensity estimate for a 128 × 128 tile and why increasing tile size has costs as well as benefits.
Why it matters: The estimate makes the benefit of reuse concrete without implying that ever-larger tiles are free.
Key terms: output tile, reuse, arithmetic intensity, BLOCK_M, BLOCK_N, BLOCK_K, accumulator, tl.dot, square output tiles, rectangular tile
5.5.5 The first tiled kernel in Triton
This section walks through a first tiled Triton matrix-multiplication kernel that computes one output tile per program under exact-divisibility assumptions.
- Output tile computation: Trace how one program selects a tile of C, builds pointer tiles for A and B, accumulates products across K slices, and stores the completed result. Explain why the accumulator uses FP32.
Why it matters: This is the calculation at the heart of the first tiled matrix-multiplication kernel. - First kernel shape restrictions: State the exact-divisibility requirement for the first tiled kernel and explain why it simplifies this implementation. Distinguish equal BLOCK_M and BLOCK_N settings from different settings.
Why it matters: These assumptions explain why this initial kernel can omit edge handling.
Key terms: C tile, pid_m, pid_n, offs_m, offs_n, K-strip, BLOCK_M, BLOCK_N, BLOCK_K, boundary masks
5.6 The Triton primitives behind the tiled kernel
Introduces the tl.dot operation that performs the main arithmetic in a tiled matrix multiplication kernel.
- Tile-level dot product accumulation: Explain that acc = tl.dot(a, b, acc) multiplies two 2D blocks and adds their product to an accumulator, connecting Triton’s tile-level code to GPU matrix hardware.
Why it matters: This operation is where the tiled kernel does most of its matrix multiplication work.
Key terms: tl.dot, 2D blocks, accumulator, tile-level programming model
5.6.1 The tl.dot block contraction
This subsection explains what tl.dot computes on matrix tiles and how a tiled kernel uses it while handling memory traversal and accumulation separately.
- Block contraction shape contract: Explain which dimensions tl.dot contracts and what shape it returns for 2D and batched 3D inputs. Apply the matching-dimension rule to identify a valid or invalid pair of input tiles.
Why it matters: The shape contract tells you whether two operand tiles can be multiplied and what output tile they produce. - Tile operation and accumulator: Describe tl.dot as a block matrix multiply-accumulate and distinguish its high-level tile operation from a sequence of scalar multiply-adds.
Why it matters: Expressing the work as a block operation gives the compiler the matrix-level operation it needs to organize execution. - K loop memory boundary: Trace how successive K slices contribute to an output tile and distinguish tl.dot's computation from the surrounding kernel's memory operations.
Why it matters: Keeping computation separate from memory traversal clarifies what tl.dot does and what the kernel author must still specify. - FP32 accumulation block: Explain why this kernel initializes an FP32 accumulator for FP16 or BF16 products and when the completed result is converted.
Why it matters: The accumulator's dtype determines how partial products are retained across the K loop.
Key terms: tl.dot, contracted dimension, BLOCK_K, BLOCK_M, BLOCK_N, batch dimension, block matrix operation, accumulator, matrix multiply-accumulate, K slice
5.6.2 Describing tiles with tl.make_block_ptr
This section shows how tl.make_block_ptr describes matrix-multiplication tiles without manually constructing pointer arrays and masks.
- Structured block-pointer descriptors: Explain how tl.make_block_ptr replaces manual pointer-array and mask setup with a descriptor for an N-dimensional memory block. Describe what the descriptor records and distinguish creating it from loading data.
Why it matters: It lets you describe a tile directly instead of assembling its addresses and masks by hand. - Matrix operand tile descriptions: Describe the parent shapes, starting offsets, and block shapes specified for A and B in the matrix-multiplication example, and state what a load from either descriptor produces.
Why it matters: These descriptions identify the two operand blocks supplied to the tiled multiplication.
Key terms: tl.make_block_ptr, block-pointer descriptor, shape, strides, starting offset, block shape, A, B, BLOCK_M, BLOCK_K
5.6.3 Advancing tiles with tl.advance
Explains how tl.advance moves block-pointer descriptors through the shared K dimension of an aligned tiled matrix multiplication, and how that traversal compares with manual pointers.
- Block pointer advancement semantics: Explain what tl.advance changes in a block-pointer descriptor, how its offset tuple is interpreted, and why the returned descriptor is reassigned in a loop.
Why it matters: This is the distinction between moving a tile’s logical position and changing the tensor it describes. - Matching K tile movement: Trace how the A and B block pointers advance after each tl.dot so the next operation receives matching slices of their shared K dimension.
Why it matters: Both inputs must advance together along K for each dot product to combine the corresponding slices. - Descriptor versus manual traversal: Distinguish the aligned block-pointer kernel’s memory traversal from the manual-pointer kernel’s while identifying the calculation they share.
Why it matters: The comparison isolates what block pointers simplify without suggesting they change the matrix multiplication. - Aligned loads and boundary handling: Explain why the aligned block-pointer kernel needs no boundary checks, and distinguish block-pointer load boundary handling from ordinary masked pointer-block loads.
Why it matters: Moving a descriptor is not a bounds check, and the aligned example deliberately avoids edge-tile handling. - Aligned host wrapper contract: State the input and shape restrictions enforced by the aligned host wrappers and describe the relationship between accumulation and output dtype.
Why it matters: The wrapper enforces the assumptions that let these aligned kernels omit edge handling.
Key terms: tl.advance, descriptor, offset tuple, element offsets, shared K dimension, BLOCK_K, a_block_ptr, b_block_ptr, tl.dot, manual-pointer kernel
5.7 Why tiled matrix multiplication uses Tensor Cores
This section explains why a tiled Triton matrix multiplication can use Tensor Cores, how `tl.dot` relates to native matrix instructions, and how those instructions fit into the rest of a kernel.
- Scalar versus matrix arithmetic: Explain why Tensor Cores are suited to tiled matrix multiplication rather than expressing the same work as scalar multiplies, adds, and reductions. Describe MMA as D = AB + C and account for its multiplication and accumulation work.
Why it matters: It connects the tiled algorithm to the hardware advantage it is designed to use. - Cooperative native matrix operations: Distinguish a Triton program tile from a native matrix instruction, and describe how threads collectively supply matrix operands and receive results.
Why it matters: It prevents confusing the block operation written in Triton with a single hardware instruction or one thread's work. - Conditional Tensor Core compilation: Explain what `tl.dot` tells Triton's compiler and why it does not guarantee a particular Tensor Core instruction.
Why it matters: It clarifies the boundary between the operation a kernel requests and the hardware instructions a compiler chooses. - Shared kernel execution resources: Trace how one tiled K-loop iteration uses different SM resources, and explain why Tensor Cores can still wait for operands.
Why it matters: It shows why using Tensor Cores does not remove the kernel's pointer, memory, and operand-supply work.
Key terms: CUDA cores, Tensor Cores, FMA, MMA, warp scheduler, program tile, native matrix shape, fragments, tl.dot, target backend
5.8 From aligned tiles to an optimized matrix multiplication
This section identifies why the earlier tiled matrix-multiplication kernels are not yet general before introducing the improvements to come.
- Limits of earlier tiled kernels: Explain why the earlier tiled matrix-multiplication kernels are not yet general: they require dimensions to end at tile boundaries and use the same tile order and launch configuration for every shape.
Why it matters: These assumptions explain why the learning kernels need further work for varied matrix shapes.
Key terms: tiled kernels, tile boundary, tile order, launch configuration
5.8.1 Handling incomplete edge tiles
Explains how masks and zero-filled input loads let a tiled matrix multiplication handle incomplete edge tiles safely.
- Incomplete K tile handling: Explain how the kernel handles a final K tile that extends beyond the matrices while still passing a complete tile to tl.dot.
Why it matters: This lets a tiled multiplication work when K is not a multiple of BLOCK_K. - Edge load and store masks: Distinguish how load and store masks handle incomplete M and N tiles, and explain why masked input loads need other=0.0.
Why it matters: Preventing invalid memory accesses alone is not enough to keep edge-tile results correct.
Key terms: K, BLOCK_K, mask, tl.dot, accumulator, M, N, load masks, store mask, other=0.0
5.8.2 Reordering output tiles to reuse L2 data
This subsection explains how grouping output tiles by program ID can improve matrix multiplication’s L2 cache reuse without changing its result.
- Grouped output tile ordering: Explain how row-major and grouped program-ID mappings visit output tiles, and why grouped ordering can improve input-tile reuse.
Why it matters: Nearby programs may reuse an input tile from L2 rather than fetch it again from HBM. - Logical mapping and measured performance: Distinguish program-ID mapping from GPU execution order, and explain why the benefit of grouped mapping must be measured.
Why it matters: A promising tile order is not a guaranteed speedup on every matrix or GPU.
Key terms: row-major mapping, grouped mapping, GROUP_M, L2 cache, HBM, A tile, program IDs, matrix shape, tile sizes
5.8.3 Letting the autotuner choose a launch configuration
This section explains how Triton selects a matrix-multiplication launch configuration for each shape and why benchmarks must account for tuning time.
- Shape-dependent launch choices: Explain why one launch configuration is not best for every matrix and why Triton measures several candidates instead of relying on one hand-picked choice.
Why it matters: It explains why the kernel needs shape-dependent tuning rather than a single fixed setup. - Autotune candidates and keys: Describe what a triton.Config candidate contains and how triton.autotune selects a configuration when M, N, or K changes.
Why it matters: It connects the tunable launch parameters to when Triton measures and reuses a choice. - Steady-state benchmark warmup: Explain why a steady-state benchmark must warm up an autotuned kernel before timing it.
Why it matters: Warmup keeps tuning overhead from being mistaken for the kernel’s steady-state runtime.
Key terms: launch configuration, N tile, autotuner, triton.Config, BLOCK_M, BLOCK_N, BLOCK_K, num_warps, num_stages, first launch
5.8.4 Adding compiler hints carefully
Explains what compiler hints tell the matrix-multiplication compiler, why their promises must be true, and why any performance benefit must be measured.
- Compiler hint meanings: Explain that compiler hints state facts about the existing matrix multiplication rather than changing its computation. Describe the facts supplied by tl.assume, tl.multiple_of, and tl.max_contiguous and how the compiler may use them.
Why it matters: Hints help the compiler exploit layout facts without changing the calculation. - Hint validity and benchmarking: Justify why the hints are valid for this kernel and why their performance effect still needs benchmarking.
Why it matters: An unsupported hint risks correctness, while a supported hint earns its place only if measurement shows a benefit.
Key terms: compiler hints, tl.assume, tl.multiple_of, tl.max_contiguous, M and N offsets, promises, runtime checks, host wrapper, positive strides, benchmark
5.8.5 Final optimized matrix multiplication kernel
Summarizes how the final optimized matrix multiplication kernel assigns, computes, and protects output tiles, and how its host wrapper validates inputs and launches it.
- Final kernel job sequence: Explain how the decorator, grouped mapping, masks, and K loop each contribute to the final matrix multiplication kernel. State why the six launch configurations are a starting set rather than universal choices.
Why it matters: These roles provide a concise way to explain what the final kernel does without treating one configuration as best everywhere. - Wrapper checks and launch timing: Describe the inputs and compatibility checks enforced by the host wrapper, what it passes to the kernel, and when the launch grid is computed.
Why it matters: The wrapper makes the kernel’s input assumptions explicit and gives the grid the block sizes it needs.
Key terms: launch configuration, grouped mapping, masks, K loop, tl.dot, wrapper, contiguous GPU matrices, FP16, BF16, inner dimensions
5.8.6 Checking correctness across shapes and dtypes
This section explains how to check the optimized matrix multiplication kernel against an independent reference across shapes and dtypes while allowing for floating-point differences.
- Independent correctness reference: Explain why the final kernel is compared with a separate PyTorch calculation rather than only with an earlier custom kernel.
Why it matters: An independent reference makes a shared implementation error less likely to go unnoticed. - Shape and dtype coverage: Describe the shape and dtype cases used to challenge the final kernel's correctness.
Why it matters: Varied shapes and dtypes test cases that an aligned, single-dtype example would miss. - Floating point comparison tolerance: Explain why the Triton and PyTorch results are compared with dtype-specific tolerance rather than bit-for-bit equality.
Why it matters: A tolerance-based comparison can accept expected rounding differences without requiring identical bits.
Key terms: PyTorch calculation, reference, indexing mistake, M, N, K, remainders, FP16, BF16, FP32
5.9 Benchmarking the matrix multiplication progression
This section benchmarks the matrix-multiplication kernels and uses the results to compare their performance across shapes and on a roofline plot.
- Benchmark measurement basis: Describe what the benchmark measures and how its throughput figures are reported.
Why it matters: These measurement choices determine what the reported times and throughput actually compare. - Aligned kernel performance progression: Explain the measured progression from scalar matrix multiplication through tiled and optimized kernels on the aligned 4096 case.
Why it matters: The progression separates the substantial gain from tiling from the smaller effect of changing pointer representation alone. - Shape-dependent PyTorch comparison: Describe how the final kernel compares with torch.matmul across the tested shapes and dtypes, and qualify the reach of that comparison.
Why it matters: A result near PyTorch performance on one shape does not imply the same result on every workload. - Conservative roofline placement: Explain why the optimized kernel’s roofline marker begins at a lower bound for operational intensity rather than at a measured intensity.
Why it matters: The lower bound supports a meaningful roofline comparison without presenting an unmeasured intensity as an exact value.
Key terms: warm-up, medians, 2MNK operation count, scalar tl.sum kernel, tiling, block-pointer version, grouped ordering, autotuning, torch.matmul, FP16
5.10 Summary
Concepts explained in this chapter
- strideThe number of storage elements between successive logical elements along an array or tensor dimension.
- memory layoutMemory layout is the mapping between logical data structures and their locations in memory.
- tl.dottl.dot is a Triton operation that computes the dot product of compatible input blocks, typically as matrix multiplication.
- memory coalescingMemory coalescing is an access pattern in which neighboring parallel threads access adjacent memory locations so those accesses can be combined efficiently.
- matrix-matrix multiplicationMatrix multiplication combines two compatible matrices by taking row-column dot products.
- row-major orderRow-major order is a memory layout in which the elements of each row are stored contiguously before the next row.
- column-major orderColumn-major order is a memory layout in which the elements of each column are stored contiguously before the next column.
- block pointerA block pointer is a Triton descriptor containing a tensor’s base address, shape, strides, block shape, and offsets for accessing a tensor region.
- tiled transposeA tiled transpose rearranges a matrix tile so its elements are written in transposed order while preserving efficient memory access.
- broadcast operationBroadcasting makes tensor values behave as though they were repeated across compatible dimensions without necessarily copying the underlying data.
- maskA mask is a per-element boolean predicate that enables or suppresses selected elements of a GPU memory operation.
- matrix multiply-accumulateA matrix multiply-accumulate operation multiplies two matrices and adds a third matrix to the product.
- arithmetic intensityArithmetic intensity is the ratio of floating-point operations performed to bytes transferred between a kernel and global memory.
- matrix multiplication kernelA matrix multiplication kernel is a GPU kernel that computes a matrix product.
- transpositionA transpose exchanges a matrix’s rows and columns or, more generally, permutes a tensor’s dimensions.
- Tensor CoreA Tensor Core is a specialized GPU execution unit for high-throughput matrix and tensor operations.
- viewAn alternative shape or indexing interpretation of existing tensor storage that does not copy its data.
- tl.advancetl.advance is a Triton operation that returns a block-pointer descriptor with its offsets advanced while preserving its other fields.
- tileA tile is a portion of a tensor processed by one Triton program instance.
- benchmarkingA benchmark is a specified test or measurement procedure used to evaluate program performance.
- autotuningAutotuning is the process of evaluating alternative implementation configurations and selecting one based on measured performance.
- tl.make_block_ptrtl.make_block_ptr is a Triton API function that creates a descriptor for a block of memory from its base address, shape, strides, offset, block shape, and memory order.
- roofline modelA roofline model is a performance model that bounds a kernel's achievable throughput using arithmetic intensity, peak compute throughput, and peak memory bandwidth.
- TritonThe Triton compiler transforms Triton programs into optimized GPU code for a target backend.
- GPU kernelA GPU kernel is a function launched on a GPU to perform computation in parallel.
- tile shapeTile shape specifies the dimensional arrangement of elements in a tile.
- offsetAn offset is an index displacement used to address an element within a data region.
- floating-point operation countA floating-point operation count is the total number of floating-point operations required by an algorithm or program.
- throughputThroughput is the amount of computation or data processing completed per unit of time.
- L2 cacheAn L2 cache is a hardware-managed GPU cache that is generally larger and farther from compute units than the L1 cache.
- element-wise multiplicationAn elementwise product multiplies corresponding elements of compatible arrays.
- gridA GPU grid is the complete collection of blocks or program instances launched for one kernel call.
- tl.sumtl.sum is a Triton operation that reduces elements along a specified dimension and can select an accumulator data type.
- effective memory bandwidthThe achieved rate of data transfer during a computation, as distinct from the memory system's peak bandwidth.
- tensorA tensor is a multidimensional array used to represent and manipulate structured data.
- tl.program_idA program ID is an integer identifying a Triton program instance along a selected dimension of the launch grid.
- warming upA warmup run is an untimed execution performed before measurement to complete initialization and bring hardware into a representative operating state.
- tl.loadtl.load is Triton's operation for loading values from memory through pointer expressions, with optional per-element masking.
- tl.storetl.store is Triton's operation for storing values to memory through pointer expressions, with optional per-element masking.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
