Chapter 3: Fused kernels and parallel patterns
The introduction recalls the independent per-program pattern of earlier Triton kernels and frames the move toward more complex workloads.
After this chapter you will be able to
- Independent program instance pattern. Describe how the earlier Triton kernels divided work among program instances: each instance loaded its own input slice, computed on it, and stored the result without needing to coordinate with other instances.
- Memory Wall Performance Gap. Explain why a sequence of simple machine-learning operations can underperform on a GPU despite looking clean in Python and offering plenty of parallel computation.
- Independent elementwise subtasks. Explain why element-wise addition is embarrassingly parallel: each output uses the inputs at its own index and can be computed without information from other outputs.
- SiLU element independence. Explain why SiLU is embarrassingly parallel: each output is its input multiplied by that input’s sigmoid, with no dependency on other elements.
- Separate kernel memory round trips. Trace the global-memory traffic for `SiLU(x) * y` when its two elementwise operations run as separate kernels, and explain why the intermediate round trip is costly.
- Kernel arithmetic intensity. Explain arithmetic intensity as the number of floating-point operations a kernel performs per byte transferred to and from global memory, and use it to describe the balance between computation and data movement.
- Low-intensity performance limit. Explain why a kernel doing far fewer FLOPs per byte than the hardware’s compute-to-bandwidth ratio can be limited by memory bandwidth rather than arithmetic throughput, using the H100 example as an illustration.
- Memory and compute bottlenecks. Explain how a kernel’s arithmetic intensity helps distinguish memory-bound from compute-bound execution, including what limits throughput in each case and representative operations.
- Roofline axes and ceilings. Describe what the roofline model plots and what its compute and memory roofs represent.
- Bottlenecks and optimization headroom. Explain how a kernel’s position on a roofline chart indicates its bottleneck and its remaining optimization headroom.
- Five-level GPU memory hierarchy. Describe the five memory levels in order from fastest and smallest to slowest and largest, and explain that they are not interchangeable.
- Data location and kernel stalls. Explain how accessing registers rather than global memory affects waiting time and why global-memory trips can slow kernels.
- Private registers and arithmetic. Explain that each GPU thread has private registers that hold active values and feed arithmetic operations directly.
- Finite register capacity and fusion. Explain how increased register use per thread can reduce concurrent threads and why fusing more operations can cause this.
- Occupancy and performance balance. Define occupancy and explain why maximizing it alone does not guarantee a fast kernel.
- Block-wide data sharing. Explain how shared memory differs from registers and how threads use it to cooperate.
- Transparent cache operation. Explain how GPU caches differ from shared memory and why a repeated global-memory access may be faster.
- Central role of L2. Trace which memory traffic passes through L2 and explain why its bandwidth matters.
- Latency Versus Aggregate Bandwidth. Distinguish the wait for one memory request from the total data rate of many requests, and explain how GPUs tolerate that wait.
- Fusion Avoids DRAM Round Trips. Explain why fusing two operations can be much faster than passing their intermediate result through separate kernels.
- Fused kernel data flow. Explain how one fused kernel computes x * σ(x) * y without storing the SiLU intermediate in global memory, and why this reduces memory traffic without approximating the computation.
- Memory traffic and speedup. Compare the per-element global-memory accesses of the unfused and fused float32 computations, and explain when their traffic ratio suggests a speedup.
- Intermediate materialization cost. Explain how a naive SwiGLU implementation moves its intermediate projections through global memory and why that makes the elementwise portion memory-bound in the section's example.
- Register resident SwiGLU fusion. Trace how the illustrated Triton kernel reads gate and value data, computes SiLU and their product in registers, and avoids storing the activation intermediate.
- Conditional logic and warp divergence. Explain how data-dependent operations differ from the straight-line kernels shown so far, and why an if-else branch can pose a performance problem on an NVIDIA GPU.
- Divergent branch execution cost. Trace what happens when threads in one warp choose different if/else branches, using the ReLU example, and explain when that serialization hurts performance.
- Per-lane value selection. Explain how tl.where uses a boolean mask to choose between two candidate values for each lane while keeping the warp on one instruction path.
- Candidate computation cost. Explain why a cheap tl.where selection does not necessarily make a conditional computation cheap.
- Logical coordinates and flat memory. Explain why a multidimensional tensor needs indexing arithmetic to access GPU memory, and describe how the earlier vector-addition kernel calculated its offsets.
- Coordinate mapping for mismatched layouts. Explain what coordinate mapping means and why operations whose computation does not align neatly with data layout need it. Describe RoPE only at the high level given here.
- Paired feature rotations. Describe how RoPE pairs feature values and rotates each pair in query and key vectors according to its sequence position. Explain how pair-specific frequencies make some pairs rotate faster than others.
- Relative position in attention. Explain why the dot product of a RoPE-rotated query and key depends on the difference between their positions rather than on their absolute positions separately.
- Split feature pairs and frequencies. Explain how the LLaMA-style split layout pairs features and why its shared-frequency design needs cosine and sine values for only half the feature dimension.
- Input and cache row mapping. Describe the two row widths a program instance must map to, and explain why mixing up their offsets produces incorrect rotations.
- Paired input coordinate mapping. Trace how a sequence position and pair number become the two flat offsets for that pair in x, and distinguish those offsets from indexing the smaller cos and sin rows.
- Logical to physical coordinate mapping. Explain why a program ID does not always map directly to an array index, and how Triton simplifies—but does not eliminate—the work of translating algorithm coordinates into memory offsets.
- Indexing checks before coding. Describe how to work out and verify a kernel’s indexing before writing it, including the different strides used for RoPE’s input and auxiliary tensors.
What it builds on
- Kernel execution flow (chapter 2) · The execution flow of the first Triton kernel.
- A complete Triton kernel: vector addition end to end (chapter 2) · Uses the vector-addition example introduced in the previous chapter.
- The GPU execution model (chapter 1) · Assumes the GPU execution model.
- The launch grid (chapter 2) · The launch grid determines how many program instances run.
- How GPUs handle data (chapter 1) · The earlier overview of GPU memory as a layered pyramid.
- Loading data and performing the computation (chapter 2) · Loading data and performing computation in a Triton kernel.
- Handling edges with a mask (chapter 2) · Using a mask to handle edge elements in a Triton kernel.
- Computing the element offsets (chapter 2) · The vector-addition kernel’s element-offset calculation.
Section by section
Introduction
The introduction recalls the independent per-program pattern of earlier Triton kernels and frames the move toward more complex workloads.
- Independent program instance pattern: Describe how the earlier Triton kernels divided work among program instances: each instance loaded its own input slice, computed on it, and stored the result without needing to coordinate with other instances.
Why it matters: This simple pattern is the baseline for discussing workloads that are more complex than one independent operation per element.
Key terms: program instances, slice of the input
3.1 The memory wall and kernel fusion
This section introduces the memory wall as a reason that cleanly composed, highly parallel machine-learning operations can underperform on a GPU.
- Memory Wall Performance Gap: Explain why a sequence of simple machine-learning operations can underperform on a GPU despite looking clean in Python and offering plenty of parallel computation.
Why it matters: It explains why available GPU compute does not guarantee fast execution.
Key terms: memory wall, on-chip storage, global memory, DRAM
3.1.1 Embarrassingly parallel tasks: the baseline GPU pattern
This section uses element-wise operations to explain embarrassingly parallel work and why it maps naturally to GPU execution.
- Independent elementwise subtasks: Explain why element-wise addition is embarrassingly parallel: each output uses the inputs at its own index and can be computed without information from other outputs.
Why it matters: Independence tells you when work can be divided among GPU workers without coordination. - GPU fit for independent work: Describe why embarrassingly parallel element-wise operations fit the GPU's SIMT model: threads perform the same operation on different data without synchronization.
Why it matters: This explains why simple element-wise operations are a natural baseline for GPU programming.
Key terms: embarrassingly parallel, element-wise operations, warps, Single Instruction, Multiple Threads (SIMT)
3.1.2 SiLU as an embarrassingly parallel kernel
The section uses SiLU to show how an elementwise Triton kernel scales across independent blocks, then explains the memory cost of chaining separate kernels and clarifies floating-point performance terminology.
- SiLU element independence: Explain why SiLU is embarrassingly parallel: each output is its input multiplied by that input’s sigmoid, with no dependency on other elements.
Why it matters: Element independence explains why SiLU work can be spread across the GPU without coordination between blocks. - Block execution and grid coverage: Describe how the Triton SiLU kernel handles blocks independently and how ceiling division sets a grid large enough to cover all N elements.
Why it matters: Independent blocks and a sufficiently large grid let the kernel process the whole input, including a final incomplete block. - Separate kernel memory round trips: Trace the global-memory traffic for `SiLU(x) * y` when its two elementwise operations run as separate kernels, and explain why the intermediate round trip is costly.
Why it matters: This avoidable intermediate-memory traffic motivates the discussion of kernel fusion. - Floating-point work versus rate: Distinguish a FLOP as one floating-point operation, FLOPs as a work count, and FLOPS or FLOP/s as operations per second.
Why it matters: Separating work from work per second prevents confusion when discussing GPU performance.
Key terms: SiLU, sigmoid, embarrassingly parallel, program instance, block, registers, global memory, grid, triton.cdiv, intermediate
3.1.3 Arithmetic intensity and the memory wall
Defines arithmetic intensity and explains how the growing gap between GPU compute throughput and memory bandwidth makes data movement a limit on kernel performance.
- Kernel arithmetic intensity: Explain arithmetic intensity as the number of floating-point operations a kernel performs per byte transferred to and from global memory, and use it to describe the balance between computation and data movement.
Why it matters: It gives you a way to compare a kernel’s computation with the data it must move. - Low-intensity performance limit: Explain why a kernel doing far fewer FLOPs per byte than the hardware’s compute-to-bandwidth ratio can be limited by memory bandwidth rather than arithmetic throughput, using the H100 example as an illustration.
Why it matters: A fast arithmetic unit cannot speed up a kernel that spends its time waiting for data. - Bandwidth gap and memory wall: Distinguish arithmetic bandwidth from memory bandwidth and describe the memory wall as the widening gap between GPU compute throughput and the rate at which memory can supply data.
Why it matters: The widening gap explains why improving compute capability alone may not improve a kernel’s speed.
Key terms: arithmetic intensity, FLOPs, global memory, peak arithmetic intensity, memory bandwidth, memory bound, arithmetic bandwidth, memory wall
3.1.4 Compute-bound vs. memory-bound
This section explains how arithmetic intensity, relative to GPU hardware capability, determines whether a kernel is limited by memory bandwidth or computation.
- Memory and compute bottlenecks: Explain how a kernel’s arithmetic intensity helps distinguish memory-bound from compute-bound execution, including what limits throughput in each case and representative operations.
Why it matters: Identifying the bottleneck tells you whether memory traffic or arithmetic throughput is the more relevant performance limit.
Key terms: arithmetic intensity, memory-bound, compute-bound, memory bandwidth, HBM, ALUs
3.1.5 Roofline model
The section uses the roofline model to show both what limits a GPU kernel’s performance and how much room it has to improve.
- Roofline axes and ceilings: Describe what the roofline model plots and what its compute and memory roofs represent.
Why it matters: The axes and roofs give you the reference points needed to interpret a kernel’s performance. - Bottlenecks and optimization headroom: Explain how a kernel’s position on a roofline chart indicates its bottleneck and its remaining optimization headroom.
Why it matters: Identifying a bottleneck alone is not enough; the chart also indicates whether substantial improvement is still possible.
Key terms: roofline model, arithmetic intensity, compute roof, memory roof, memory-bound, compute-bound, optimization headroom, intersection of the two roofs
3.2 The GPU Memory System
Introduces the GPU’s five-level memory hierarchy and explains why the location of data can determine whether a kernel computes or stalls.
- Five-level GPU memory hierarchy: Describe the five memory levels in order from fastest and smallest to slowest and largest, and explain that they are not interchangeable.
Why it matters: The hierarchy gives you a way to reason about where data access costs arise. - Data location and kernel stalls: Explain how accessing registers rather than global memory affects waiting time and why global-memory trips can slow kernels.
Why it matters: A kernel can run far below the hardware’s compute capability if its threads spend too much time waiting for data.
Key terms: registers, shared memory, L1 cache, L2 cache, global memory, register file, HBM3 global memory, latency, bandwidth, intermediate results
3.2.1 Registers hold what matters right now
This section explains how fast, private registers support GPU computation and why their limited capacity creates performance trade-offs for Triton kernels.
- Private registers and arithmetic: Explain that each GPU thread has private registers that hold active values and feed arithmetic operations directly.
Why it matters: This explains why registers are useful for values a thread needs right now. - Finite register capacity and fusion: Explain how increased register use per thread can reduce concurrent threads and why fusing more operations can cause this.
Why it matters: It explains why adding work to a fused kernel can unexpectedly slow it down. - Register spilling to local memory: Describe what happens when a kernel needs more registers than are available and why the fallback can be costly.
Why it matters: It shows why simply avoiding spills can matter more than maximizing the number of running threads. - Occupancy and performance balance: Define occupancy and explain why maximizing it alone does not guarantee a fast kernel.
Why it matters: It gives you a way to discuss concurrent work without treating one metric as the whole performance story.
Key terms: registers, Arithmetic Logic Units (ALUs), register file, register usage, fused kernels, intermediate values, register spilling, local memory, L1 and L2 caches, occupancy
3.2.2 Shared memory enables cooperation
Explains how shared memory lets threads in a block cooperate, reuse and rearrange data, and how Triton handles that storage.
- Block-wide data sharing: Explain how shared memory differs from registers and how threads use it to cooperate.
Why it matters: This explains how threads can work together without each fetching the same data from global memory. - Shared memory access rearrangement: Describe how shared memory can support algorithms that exchange data and change access patterns.
Why it matters: Shared memory is useful not only for reuse but also for coordinating work and improving memory access patterns. - Shared memory cache trade-off: Explain the capacity trade-off between shared memory and L1 cache on the NVIDIA GPUs described.
Why it matters: The best partition depends on whether a kernel benefits more from explicit data placement or automatic caching. - Triton shared memory abstraction: Describe how Triton programmers encounter shared memory without explicitly declaring its buffers.
Why it matters: It clarifies why shared memory matters to Triton performance even when it is not declared in source code.
Key terms: shared memory, thread block, registers, global memory, staging area, tiles, memory access patterns, L1 cache, unified data cache, explicit data placement
3.2.3 Caches provide transparent speedup
Explains how the GPU’s L1 and L2 caches automatically speed access to global memory, and why L2 bandwidth matters.
- Transparent cache operation: Explain how GPU caches differ from shared memory and why a repeated global-memory access may be faster.
Why it matters: This explains why repeated memory accesses need not all incur a trip to DRAM. - L1 locality and latency: Describe where L1 cache resides and how the latency of an L1 hit compares with other memory levels.
Why it matters: L1’s location and relative latency help explain where a cache hit fits in the memory hierarchy. - Central role of L2: Trace which memory traffic passes through L2 and explain why its bandwidth matters.
Why it matters: L2’s central position makes its bandwidth relevant even when considering global-memory traffic.
Key terms: caches, shared memory, global memory, DRAM, L1 cache, SM, register, L2, HBM, L2 cache
3.2.4 Global memory trades capacity for latency
Explains why global memory can offer high bandwidth yet have high access latency, and why keeping intermediate results in registers makes kernel fusion valuable.
- Global Memory Access Cost: Describe global memory’s role as the GPU’s large-capacity storage and trace why an individual access can take hundreds of cycles.
Why it matters: The path to DRAM explains why a single global-memory access is expensive despite the memory’s large capacity. - Latency Versus Aggregate Bandwidth: Distinguish the wait for one memory request from the total data rate of many requests, and explain how GPUs tolerate that wait.
Why it matters: It prevents high bandwidth from being mistaken for a fast individual access. - Fusion Avoids DRAM Round Trips: Explain why fusing two operations can be much faster than passing their intermediate result through separate kernels.
Why it matters: The register-versus-DRAM cost difference is the performance reason for kernel fusion in this section.
Key terms: global memory, DRAM, SM, L2 cache, memory controller, latency, bandwidth, requests in flight, properly aligned, kernel fusion
3.3 Kernel fusion: the solution
This section explains how kernel fusion reduces global-memory traffic, quantifies the benefit for a SiLU-and-multiplication chain, and shows how the same idea applies to attention.
- Fused kernel data flow: Explain how one fused kernel computes x * σ(x) * y without storing the SiLU intermediate in global memory, and why this reduces memory traffic without approximating the computation.
Why it matters: Keeping intermediates on chip lets the GPU do the same work with less global-memory traffic. - Memory traffic and speedup: Compare the per-element global-memory accesses of the unfused and fused float32 computations, and explain when their traffic ratio suggests a speedup.
Why it matters: The access count shows both the source of fusion’s benefit and why the predicted speedup is not guaranteed. - Fusion in attention: Describe how the Flash Attention example extends fusion beyond elementwise operations to avoid materializing the full attention matrix.
Why it matters: The attention example shows that avoiding intermediate memory round trips can matter well beyond a simple activation chain.
Key terms: kernel fusion, global memory, registers, intermediate results, arithmetic intensity, memory accesses, float32, memory-bound, kernel launch overhead, occupancy
3.3.1 Case Study: SwiGLU
This section uses SwiGLU to show how fusing elementwise operations avoids materializing an intermediate result and reduces global-memory traffic.
- SwiGLU gate and value: Describe SwiGLU as an elementwise product of two projections, with the gate projection activated before it multiplies the value projection. Explain why Swish is called SiLU in the simplified case used here.
Why it matters: It identifies the elementwise work that the fused kernel performs after the projections are computed. - Intermediate materialization cost: Explain how a naive SwiGLU implementation moves its intermediate projections through global memory and why that makes the elementwise portion memory-bound in the section's example.
Why it matters: It explains why avoiding intermediate memory traffic can matter more than reducing arithmetic here. - Register resident SwiGLU fusion: Trace how the illustrated Triton kernel reads gate and value data, computes SiLU and their product in registers, and avoids storing the activation intermediate.
Why it matters: It shows concretely how chaining operations without an intermediate global-memory write reduces data movement.
Key terms: SwiGLU, gate, value, up-projection, Swish, SiLU, materialization, global memory, arithmetic intensity, memory-bound
3.4 Control flow: masking over branching
Introduces why data-dependent conditional logic is needed in GPU kernels and why branching can be problematic for lockstep warps.
- Conditional logic and warp divergence: Explain how data-dependent operations differ from the straight-line kernels shown so far, and why an if-else branch can pose a performance problem on an NVIDIA GPU.
Why it matters: This motivates treating data-dependent choices differently from ordinary CPU-style branching in GPU kernels.
Key terms: straight-line execution path, conditional logic, if-else, warp, warp divergence
3.4.1 Warp divergence
Explains why different branches within a GPU warp execute serially and how that affects performance.
- SIMT warp execution: Explain that NVIDIA groups 32 threads into a warp and executes one instruction across the warp at a time, with each thread operating on its own data.
Why it matters: The warp's shared instruction stream explains why different branch choices create a problem. - Divergent branch execution cost: Trace what happens when threads in one warp choose different if/else branches, using the ReLU example, and explain when that serialization hurts performance.
Why it matters: It explains why a data-dependent conditional can reduce GPU throughput even though many threads are active.
Key terms: warp, SIMT execution, Single Instruction, Multiple Threads, warp divergence, ReLU, branch, throughput
3.4.2 Predicated execution with tl.where
Explains how tl.where selects values per lane without divergent branching, and why candidate computation and masked memory access still matter.
- Per-lane value selection: Explain how tl.where uses a boolean mask to choose between two candidate values for each lane while keeping the warp on one instruction path.
Why it matters: This is how an element-wise condition can produce different results across lanes without divergent control flow. - Uniform and element-wise conditions: Distinguish an if condition shared by every lane in a block from an element-wise condition whose outcome differs across lanes.
Why it matters: The distinction tells you when an if statement is appropriate and when per-lane selection avoids divergence. - Candidate computation cost: Explain why a cheap tl.where selection does not necessarily make a conditional computation cheap.
Why it matters: Avoiding divergent branches does not mean the unselected candidate is free to compute. - Inactive-lane memory masking: Describe how a masked tl.load handles inactive lanes and why computations on either side of tl.where must remain safe for them.
Why it matters: Selecting an output afterward does not by itself make work on inactive lanes harmless.
Key terms: tl.where, boolean mask, per-lane select, warp, if statement, element-wise conditions, program ID, tile-level constant, warp divergence, candidate values
3.5 Advanced indexing: coordinate mapping (RoPE)
The section introduces coordinate mapping: turning an algorithm’s logical positions into addresses in flat GPU memory, especially when computation and data layout do not align neatly.
- Logical coordinates and flat memory: Explain why a multidimensional tensor needs indexing arithmetic to access GPU memory, and describe how the earlier vector-addition kernel calculated its offsets.
Why it matters: It explains why even a simple Triton load depends on calculating the right memory location. - Coordinate mapping for mismatched layouts: Explain what coordinate mapping means and why operations whose computation does not align neatly with data layout need it. Describe RoPE only at the high level given here.
Why it matters: It provides the reason more complex kernels must calculate read and write locations rather than assume they follow the computation’s order.
Key terms: GPU memory, indexing arithmetic, tl.load(ptr + offset), program ID, block size, coordinate mapping, RoPE, memory offsets, matrix transposition
3.5.1 Positional encoding in transformers
This section explains why transformers need positional information, how RoPE rotates paired query and key features to make attention sensitive to relative position, and what that implies for a GPU kernel.
- Need for positional information: Explain why attention alone does not capture token order and why positional encodings are added to token representations. Distinguish an absolute position signature from information about the distance between tokens.
Why it matters: It explains the problem that RoPE is designed to address. - Paired feature rotations: Describe how RoPE pairs feature values and rotates each pair in query and key vectors according to its sequence position. Explain how pair-specific frequencies make some pairs rotate faster than others.
Why it matters: The paired rotation is the operation a RoPE kernel must perform. - Relative position in attention: Explain why the dot product of a RoPE-rotated query and key depends on the difference between their positions rather than on their absolute positions separately.
Why it matters: This is why rotating by absolute position can give attention a relative-position signal. - RoPE kernel data needs: Describe the data a GPU kernel loads to apply RoPE and why each output creates an indexing challenge.
Why it matters: It connects the positional-encoding idea to the data accesses required by a GPU implementation.
Key terms: attention, positional encodings, absolute positions, relative distances, Rotary Positional Embeddings (RoPE), feature pairs, rotation angle, base frequency, multi-scale structure, dot product
3.5.2 The memory layout challenge
This section explains how RoPE’s split feature layout and shared rotation frequencies create two different row widths that a kernel must map correctly.
- Split feature pairs and frequencies: Explain how the LLaMA-style split layout pairs features and why its shared-frequency design needs cosine and sine values for only half the feature dimension.
Why it matters: The pairing explains why feature values and their rotation values come from differently sized rows. - Input and cache row mapping: Describe the two row widths a program instance must map to, and explain why mixing up their offsets produces incorrect rotations.
Why it matters: Keeping the two row widths distinct prevents a kernel from combining features with the wrong cached rotation values.
Key terms: split layout, Shared-Frequency design, cos and sin tensors, pid, sequence position m, input row, cached table row, strides, offset calculation
3.5.3 Building the kernel
This section builds a shared-frequency RoPE kernel by mapping sequence positions and feature pairs to the correct input and angle offsets.
- Row and pair assignment: Describe how each program instance handles one sequence position and uses local indices to identify the feature pairs in that row.
Why it matters: This assignment gives every program a row and every valid local index a pair to process. - Paired input coordinate mapping: Trace how a sequence position and pair number become the two flat offsets for that pair in x, and distinguish those offsets from indexing the smaller cos and sin rows.
Why it matters: Correct coordinate mapping fetches both members of each pair and their angles from tensors with different row widths. - Grid bounds for rows: Explain why this kernel requires exactly seq_len program instances as written and how to guard an oversized grid before loading.
Why it matters: The row guard prevents out-of-range programs from reading beyond the input. - Shared angle load economy: Explain why the shared-frequency design loads fewer auxiliary values than loading independent sine and cosine values for each input element.
Why it matters: Sharing angles across a pair reduces the amount of auxiliary data the kernel has to fetch.
Key terms: pid, half_dim, local_idx, BLOCK_SIZE, mask, coordinate mapping, row-major, row_start, first_offset, second_offset
3.5.4 Why coordinate mapping matters
Uses RoPE to show why GPU kernels need explicit coordinate-to-memory mapping and a way to check that mapping before coding.
- Logical to physical coordinate mapping: Explain why a program ID does not always map directly to an array index, and how Triton simplifies—but does not eliminate—the work of translating algorithm coordinates into memory offsets.
Why it matters: Correct arithmetic is not enough if a kernel reads from or writes to the wrong memory locations. - Indexing checks before coding: Describe how to work out and verify a kernel’s indexing before writing it, including the different strides used for RoPE’s input and auxiliary tensors.
Why it matters: Checking offsets against the grid before coding helps catch missing or conflicting writes.
Key terms: coordinate mapping, memory offsets, block-level offsets, program ID, program instance, stride, feature_dim, cos, sin
3.6 Exercises
3.6.1 Know thy GPU
3.6.2 Value clipping
3.6.3 Full elementwise RoPE
3.7 Summary
Where this is used later
- The mechanics of reduction (chapter 4) · The independence of embarrassingly parallel elementwise tasks.
- Arithmetic intensity and the roofline view (chapter 5) · Provides the arithmetic-intensity distinction for discussing matrix multiplication performance.
- Benchmarking the matrix multiplication progression (chapter 5) · The roofline model.
- Staging the transpose through shared memory (chapter 5) · Staging a transpose through shared memory draws on block-wide sharing.
- Reusing values across output elements (chapter 5) · Reusing values across output elements draws on the same data-reuse idea.
- (chapter intro) (chapter 4) · Shared memory enables cooperation within a block.
- RMS normalization in Triton (chapter 4) · Shared memory as on-chip storage.
- Triton’s primitives for reduction (chapter 4) · Triton reduction primitives provide a later context for compiler-managed exchange of partial results.
Concepts explained in this chapter
- arithmetic intensityArithmetic intensity is the ratio of floating-point operations performed to bytes transferred between a kernel and global memory.
- memory bandwidthMemory bandwidth is the rate at which a memory system transfers data.
- GPU memoryVideo memory is memory dedicated to storing data for access by a graphics processing unit.
- kernel fusionKernel fusion is an optimization technique that combines multiple operations into fewer GPU kernel launches to reduce intermediate data movement and launch overhead.
- hardware registerA register is a small, private storage location associated with a GPU thread for operands, intermediate results, and other active values.
- memory boundA memory-bound regime is a performance regime in which data-transfer capacity limits execution more than arithmetic throughput does.
- coordinate mappingCoordinate mapping is the translation from an algorithm's multidimensional coordinates to the memory addresses of the corresponding data.
- memory wallThe memory wall is a performance limitation that occurs when data movement between memory and computation is slower than the processor's ability to perform arithmetic.
- SwishSwish is a smooth activation function defined as x multiplied by sigmoid(βx), with β controlling its shape.
- maskA mask is a per-element boolean predicate that enables or suppresses selected elements of a GPU memory operation.
- memory latencyThe time between requesting a memory access and receiving its data.
- Rotary Position EmbeddingRotary Positional Embeddings encode position by applying position-dependent rotations to pairs of neural-network features.
- warp divergenceWarp divergence is the execution of different conditional branches by threads in the same warp, requiring the branches to be processed sequentially.
- compute-boundA property of a computation whose execution rate is limited primarily by the speed of arithmetic calculation rather than memory transfer.
- throughputThroughput is the amount of computation or data processing completed per unit of time.
- offsetAn offset is an index displacement used to address an element within a data region.
- register fileA register file is the hardware storage array containing the registers available to a processor execution unit.
- strideThe number of storage elements between successive logical elements along an array or tensor dimension.
- element-wise operationAn element-wise tensor operation applies the same function independently to individual tensor elements.
- embarrassingly parallelAn embarrassingly parallel task is a computation whose output elements depend independently on separate inputs and require no coordination between work instances.
- FlashAttentionFlashAttention is an IO-aware exact attention algorithm that tiles computation to avoid materializing the full attention matrix in global memory.
- occupancyGPU occupancy is the proportion of a processor's available concurrent threads or warps that are active for execution.
- L1 cacheAn L1 cache is a small, hardware-managed cache close to the GPU compute units.
- threadA CPU thread is an independently schedulable sequence of instructions executed by a CPU.
- warpA warp is a hardware group of threads that an NVIDIA streaming multiprocessor schedules and issues together.
- shared memoryShared memory is explicitly managed on-chip GPU memory visible to all threads in a single block.
- shared-frequency designA shared-frequency design uses one sine and cosine value for each pair of input elements rather than independent values for every element.
- roofline modelA roofline model is a performance model that bounds a kernel's achievable throughput using arithmetic intensity, peak compute throughput, and peak memory bandwidth.
- tl.program_idA program ID is an integer identifying a Triton program instance along a selected dimension of the launch grid.
- SwiGLUSwiGLU is a gated neural-network activation that multiplies a SiLU-activated linear projection by a second linear projection.
- element-wise fusionElement-wise fusion combines multiple element-wise tensor operations into one GPU computation.
- element-wise multiplicationAn elementwise product multiplies corresponding elements of compatible arrays.
- epilogue fusionEpilogue fusion combines operations applied after a primary computation into the same kernel so intermediate results remain available without a memory round trip.
- materializationMaterialization is the process of writing an intermediate computational result to memory so that a later operation can read it.
- ridge pointA ridge point is the arithmetic-intensity threshold separating memory-bound and compute-bound performance on a roofline model.
- streaming multiprocessorA streaming multiprocessor is NVIDIA's basic GPU computational building block that executes groups of threads and provides computational resources.
- tl.wheretl.where is a Triton operation that selects values elementwise according to a condition without requiring divergent branch execution.
- floating-point operation countA floating-point operation count is the total number of floating-point operations required by an algorithm or program.
- floating-point operations per secondFloating-point operations per second measures the rate at which a processor or system performs floating-point arithmetic.
- sigmoidThe sigmoid function commonly refers to the logistic function, which maps a real-valued input to 1 divided by 1 plus the exponential of its negation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
