What you will learn in GPU Programming with Triton
Accelerate AI training and inference
Chapter by chapter
Chapter 1: Introduction to Triton
The introduction explains why growing AI compute demands make GPUs important, why Triton fills a gap between ML frameworks and CUDA, and how combining operations in one kernel can reduce overhead.
What you learn in chapter 1 →Chapter 2: Write your first kernel in Triton
Introduces the chapter’s plan to build, examine, launch, and benchmark a vector-addition kernel in Triton.
What you learn in chapter 2 →Chapter 3: Fused kernels and parallel patterns
The introduction recalls the independent per-program pattern of earlier Triton kernels and frames the move toward more complex workloads.
What you learn in chapter 3 →Chapter 4: Reduction patterns
Introduces reductions as GPU collective operations that turn distributed values into a shared statistic through coordinated communication.
What you learn in chapter 4 →Chapter 5: Higher-dimensional kernels
The chapter introduction explains why matrix multiplication matters in deep learning and why matrix kernels require thinking about both logical dimensions and linear memory.
What you learn in chapter 5 →Concepts explained in this book
- TritonThe Triton compiler transforms Triton programs into optimized GPU code for a target backend.
- GPU kernelA GPU kernel is a function launched on a GPU to perform computation in parallel.
- program instanceA program instance is Triton's unit of work corresponding to one block of input elements in a compiled kernel launch.
- GPUA GPU device is the complete graphics-processing hardware containing the streaming multiprocessors and their connected memory system.
- hardware registerA register is a small, private storage location associated with a GPU thread for operands, intermediate results, and other active values.
- atomicityAn atomic operation performs a memory read-modify-write update as one indivisible transaction that cannot be interleaved at the target address.
- tl.atomic_addtl.atomic_add is a Triton API operation that atomically adds a value to a global-memory location and returns its previous value.
- @triton.jit@triton.jit is a Triton decorator that marks a Python function for JIT compilation into a GPU kernel.
- gridA GPU grid is the complete collection of blocks or program instances launched for one kernel call.
- GPU memoryVideo memory is memory dedicated to storing data for access by a graphics processing unit.
- threadA CPU thread is an independently schedulable sequence of instructions executed by a CPU.
- streaming multiprocessorA streaming multiprocessor is NVIDIA's basic GPU computational building block that executes groups of threads and provides computational resources.
- reduction operationA reduction combines multiple values into fewer values using an associative operation such as a sum, maximum, or minimum.
- maskA mask is a per-element boolean predicate that enables or suppresses selected elements of a GPU memory operation.
- strideThe number of storage elements between successive logical elements along an array or tensor dimension.
- arithmetic intensityArithmetic intensity is the ratio of floating-point operations performed to bytes transferred between a kernel and global memory.
- CUDACUDA is NVIDIA's parallel computing platform and programming model, including the CUDA C++ language extensions for GPU programming.
- block-level programmingA block-level programming language lets a programmer express a GPU kernel from the perspective of one block processing one tile.
- thread blockA GPU block is a group of threads scheduled as a unit that can cooperate through shared memory.
- warpA warp is a hardware group of threads that an NVIDIA streaming multiprocessor schedules and issues together.
- kernel fusionKernel fusion is an optimization technique that combines multiple operations into fewer GPU kernel launches to reduce intermediate data movement and launch overhead.
- benchmarkingA benchmark is a specified test or measurement procedure used to evaluate program performance.
- tl.program_idA program ID is an integer identifying a Triton program instance along a selected dimension of the launch grid.
- memory layoutMemory layout is the mapping between logical data structures and their locations in memory.
- SIMTThe Single Instruction, Multiple Threads (SIMT) execution model runs the same instruction across multiple threads operating on separate data.
- reduction treeA hierarchical reduction that combines values through a tree of partial results.
- tileA tile is a portion of a tensor processed by one Triton program instance.
- L1 cacheAn L1 cache is a small, hardware-managed cache close to the GPU compute units.
- shared memoryShared memory is explicitly managed on-chip GPU memory visible to all threads in a single block.
- constexprconstexpr denotes a value available at compile time rather than only at runtime.
- L2 cacheAn L2 cache is a hardware-managed GPU cache that is generally larger and farther from compute units than the L1 cache.
- memory hierarchyA memory hierarchy is an ordered arrangement of memory levels that trade capacity and access speed to keep frequently needed data near computation.
- effective memory bandwidthThe achieved rate of data transfer during a computation, as distinct from the memory system's peak bandwidth.
- tl.sumtl.sum is a Triton operation that reduces elements along a specified dimension and can select an accumulator data type.
- warming upA warmup run is an untimed execution performed before measurement to complete initialization and bring hardware into a representative operating state.
- atomic accumulationAtomic accumulation is the concurrent combination of contributions into shared output locations using atomic updates.
- block-level reductionA block-level reduction combines values within one GPU block or Triton program instance using on-chip resources and synchronization.
- CAS loopA CAS loop repeatedly reads a value, computes an update, attempts compare-and-swap, and retries when another thread changes the value.
- histogram binningThe assignment and counting of data values into discrete intervals called bins.
- implicit barrierAn implicit barrier is compiler-generated synchronization that ensures producing threads complete a stage before consuming threads use its results.
- memory_order_relaxedRelaxed memory ordering guarantees atomicity of the update without imposing ordering constraints on other memory accesses.
- memory orderingMemory ordering semantics specify which memory operations may be reordered around an atomic operation and when their effects become visible.
- scatter operationA scatter operation updates output locations selected by per-element indices or data-dependent addresses.
- split-k matrix multiplicationSplit-K matrix multiplication partitions the contracting matrix dimension across program instances that produce partial results for shared output tiles.
- tl.atomic_castl.atomic_cas is a Triton compare-and-swap operation that writes a new value only when the stored value equals an expected comparison value.
- memory contentionCompetition among concurrent operations for access to the same memory location or cache line.
- asynchronous programmingAsynchronous execution allows a processor to issue work and continue without waiting for that work to finish.
- cache flushingCache flushing is the deliberate eviction of cached data so subsequent measurements begin without data retained from earlier executions.
- CUDA eventA CUDA event is a GPU-timeline timestamp object used to record and measure points in CUDA stream execution.
- torch.cuda.synchronize()torch.cuda.synchronize() is a PyTorch CUDA API call that blocks the CPU until all previously issued GPU work has completed.
- triton.testing.assert_closetriton.testing.assert_close is a Triton testing utility that compares result and reference tensors within absolute and relative tolerances and raises an error when they differ excessively.
- triton.testing.do_benchtriton.testing.do_bench is a Triton benchmarking utility that performs warmup, synchronized timing, repetition, and runtime summarization for a callable.
- memory bandwidthMemory bandwidth is the rate at which a memory system transfers data.
- row-major orderRow-major order is a memory layout in which the elements of each row are stored contiguously before the next row.
- tl.dottl.dot is a Triton operation that computes the dot product of compatible input blocks, typically as matrix multiplication.
- hostIn GPU computing, the host is the CPU-side execution environment that launches and orchestrates device work.
- vector additionVector addition is the element-wise operation that adds corresponding components of two equal-sized vectors to produce a third vector.
- memory coalescingMemory coalescing is an access pattern in which neighboring parallel threads access adjacent memory locations so those accesses can be combined efficiently.
- memory boundA memory-bound regime is a performance regime in which data-transfer capacity limits execution more than arithmetic throughput does.
- tl.maxtl.max is a Triton operation that selects the maximum value along one or more tensor dimensions, optionally returning its index.
- memory latencyThe time between requesting a memory access and receiving its data.
- element-wise operationAn element-wise tensor operation applies the same function independently to individual tensor elements.
- embarrassingly parallelAn embarrassingly parallel task is a computation whose output elements depend independently on separate inputs and require no coordination between work instances.
- softmaxSoftmax normalization exponentiates scores and divides by their sum to produce nonnegative values that sum to one.
- materializationMaterialization is the process of writing an intermediate computational result to memory so that a later operation can read it.
- deviceIn CUDA terminology, the device is the GPU-side processor and execution environment that executes dispatched parallel work.
- coordinate mappingCoordinate mapping is the translation from an algorithm's multidimensional coordinates to the memory addresses of the corresponding data.
- cubinA cubin is a CUDA binary containing GPU machine code compiled for a particular NVIDIA GPU architecture.
- offsetAn offset is an index displacement used to address an element within a data region.
- matrix-matrix multiplicationMatrix multiplication combines two compatible matrices by taking row-column dot products.
- memory wallThe memory wall is a performance limitation that occurs when data movement between memory and computation is slower than the processor's ability to perform arithmetic.
- specializationCode specialization generates an implementation tailored to particular known argument types, shapes, or configuration values.
- global reductionA global reduction combines values distributed across all program instances in a kernel launch grid into fewer global results.
- tl.mintl.min is a Triton operation that selects the minimum value along one or more tensor dimensions, optionally returning its index.
- tl.reducetl.reduce is a Triton operation that reduces tensor values with a programmer-provided binary combine function.
- two-pass reductionA two-pass reduction first computes per-program-instance partial results and then reduces those partials in a subsequent kernel pass.
- just-in-time compilationJIT compilation translates program code into executable machine code at or near the time it is invoked.
- SwishSwish is a smooth activation function defined as x multiplied by sigmoid(βx), with β controlling its shape.
- clock cycleA clock cycle is the interval between consecutive pulses of a processor's internal clock.
- distance problemThe performance problem caused by the latency of moving data from storage to computational units.
- DRAMDRAM is a capacitor-based memory technology that requires periodic refresh and provides high storage density at higher access latency than SRAM.
- latency hidingA parallel execution technique that keeps processing units busy by running other threads while some threads wait for memory operations.
- SRAMSRAM is a transistor-based memory technology that retains state while powered and provides low-latency access without periodic refresh.
- block sizeBlock size specifies the number of elements or threads assigned to a GPU computation block or Triton program instance.
- LLVM IRLLVM IR is a typed, low-level intermediate representation used by the LLVM compiler infrastructure for analysis and code generation.
- column-major orderColumn-major order is a memory layout in which the elements of each column are stored contiguously before the next column.
- broadcast operationBroadcasting makes tensor values behave as though they were repeated across compatible dimensions without necessarily copying the underlying data.
- SPMDThe SPMD model runs the same program code across multiple program instances while each instance operates on a different portion of the data.
- Rotary Position EmbeddingRotary Positional Embeddings encode position by applying position-dependent rotations to pairs of neural-network features.
- rmsnormRMSNorm normalizes activations using their root-mean-square magnitude without subtracting the mean.
- tensorA tensor is a multidimensional array used to represent and manipulate structured data.
- block pointerA block pointer is a Triton descriptor containing a tensor’s base address, shape, strides, block shape, and offsets for accessing a tensor region.
- roofline modelA roofline model is a performance model that bounds a kernel's achievable throughput using arithmetic intensity, peak compute throughput, and peak memory bandwidth.
- matrix multiplication kernelA matrix multiplication kernel is a GPU kernel that computes a matrix product.
- tiled transposeA tiled transpose rearranges a matrix tile so its elements are written in transposed order while preserving efficient memory access.
- warp divergenceWarp divergence is the execution of different conditional branches by threads in the same warp, requiring the branches to be processed sequentially.
- compute-boundA property of a computation whose execution rate is limited primarily by the speed of arithmetic calculation rather than memory transfer.
- transpositionA transpose exchanges a matrix’s rows and columns or, more generally, permutes a tensor’s dimensions.
- communication hierarchyA communication hierarchy is an ordered set of communication mechanisms whose scope and storage locations vary across threads, warps, blocks, and the grid.
- PTXPTX is NVIDIA's low-level virtual instruction-set representation for GPU programs.
- autotuningAutotuning is the process of evaluating alternative implementation configurations and selecting one based on measured performance.
- matrix multiply-accumulateA matrix multiply-accumulate operation multiplies two matrices and adds a third matrix to the product.
- CPUA central processing unit is a general-purpose processor optimized for low-latency sequential and moderately parallel execution.
- data parallelismData parallelism applies the same computation independently to multiple partitions of data.
- latency optimizationA latency-optimized design minimizes the time required to complete an individual task or thread of execution.
- throughput-optimized designA throughput-optimized design maximizes the amount of work completed over a given period, often by executing many tasks concurrently.
- triton.cdivtriton.cdiv is a Triton helper that computes integer ceiling division.
- throughputThroughput is the amount of computation or data processing completed per unit of time.
- GPU runtimeA GPU runtime is the software layer that provides device memory management, kernel dispatch, and execution support for a GPU backend.
- reduce-then-mapA computation pattern that first reduces values to a statistic and then uses that statistic to transform each element.
- Tensor CoreA Tensor Core is a specialized GPU execution unit for high-throughput matrix and tensor operations.
- vectorizationVectorization is an optimization technique that performs one operation on multiple data elements using vector or SIMD instructions.
- FlashAttentionFlashAttention is an IO-aware exact attention algorithm that tiles computation to avoid materializing the full attention matrix in global memory.
- occupancyGPU occupancy is the proportion of a processor's available concurrent threads or warps that are active for execution.
- element-wise multiplicationAn elementwise product multiplies corresponding elements of compatible arrays.
- parallelismParallelism is the simultaneous execution of multiple tasks by multiple physical or computational resources.
- concurrencyConcurrency is the organization of multiple tasks so that their progress overlaps, often by interleaving work rather than executing tasks simultaneously.
- floating-point operation countA floating-point operation count is the total number of floating-point operations required by an algorithm or program.
- tile shapeTile shape specifies the dimensional arrangement of elements in a tile.
- shared-frequency designA shared-frequency design uses one sine and cosine value for each pair of input elements rather than independent values for every element.
- viewAn alternative shape or indexing interpretation of existing tensor storage that does not copy its data.
- domain-specific languageA domain-specific language is a programming language designed for a particular class of problems.
- floating-point additionFloating-point addition is addition performed on finite-precision floating-point representations with rounding of the computed result.
- block-based languageA block-based language expresses computation over whole data blocks rather than individual scalar threads.
- embedded domain-specific languageAn embedded domain-specific language is a domain-specific language expressed within a host programming language.
- hardware portabilityHardware portability is the ability to execute the same program source across different hardware targets through backend-specific compilation.
- ROCmROCm is AMD's open-source GPU computing stack comprising driver, runtime, and compiler-toolchain components.
- asm attributeThe asm attribute is a compiled Triton kernel object's mapping from compilation-stage names to generated source or binary representations.
- Compiler ExplorerCompiler Explorer is an online tool that compiles source code and displays its generated intermediate and target representations.
- intermediate representationAn intermediate representation is a structured compiler artifact that expresses a program between its source form and its final executable form.
- ir inspectionIR inspection is the examination of compiler intermediate representations to understand program lowering and generated code.
- accumulation precisionHigher-precision accumulation performs an aggregate computation in a numerical format with greater precision than the input format.
- associativityAssociativity is the algebraic property that regrouping operands does not change an operation's result.
- stable reformulationA stable reformulation rewrites a numerical computation to reduce amplification of overflow, underflow, or cancellation errors.
- tl.advancetl.advance is a Triton operation that returns a block-pointer descriptor with its offsets advanced while preserving its other fields.
- register fileA register file is the hardware storage array containing the registers available to a processor execution unit.
- pointerA pointer is a variable that stores a memory address used to locate data in memory.
- tl.constexprtl.constexpr is a Triton annotation that marks a kernel parameter as a compile-time constant for specialization.
- type hintsA type annotation is syntax attached to a programming-language parameter or value that specifies type-related metadata or compilation semantics.
- tl.argmaxtl.argmax is a Triton operation that returns the index of a maximum value along a tensor dimension.
- tl.argmintl.argmin is a Triton operation that returns the index of a minimum value along a tensor dimension.
- element-wise fusionElement-wise fusion combines multiple element-wise tensor operations into one GPU computation.
- epilogue fusionEpilogue fusion combines operations applied after a primary computation into the same kernel so intermediate results remain available without a memory round trip.
- ridge pointA ridge point is the arithmetic-intensity threshold separating memory-bound and compute-bound performance on a roofline model.
- SwiGLUSwiGLU is a gated neural-network activation that multiplies a SiLU-activated linear projection by a second linear projection.
- numerical overflowA numerical condition in which a computed value exceeds the range representable by its number format.
- tl.loadtl.load is Triton's operation for loading values from memory through pointer expressions, with optional per-element masking.
- tl.storetl.store is Triton's operation for storing values to memory through pointer expressions, with optional per-element masking.
- tl.wheretl.where is a Triton operation that selects values elementwise according to a condition without requiring divergent branch execution.
- tl.arangetl.arange is a Triton operation that constructs a sequence of regularly spaced integer offsets over a specified range.
- precisionThe fineness with which a numerical representation distinguishes nearby values.
- floating-point operations per secondFloating-point operations per second measures the rate at which a processor or system performs floating-point arithmetic.
- sigmoidThe sigmoid function commonly refers to the logistic function, which maps a real-valued input to 1 divided by 1 plus the exponential of its negation.
- fused softmax kernelA fused softmax kernel computes the softmax reductions and elementwise operations in one GPU kernel while retaining intermediate values on chip.
- inverse RMSThe inverse RMS is the reciprocal of a vector's root mean square and serves as its normalization factor.
- learned scale vectorA learned scale vector is a trainable vector of per-element multiplicative factors applied to normalized activations.
- root mean squareThe root mean square is the square root of the arithmetic mean of squared values.
- two-pass computationTwo-pass computation performs one pass to calculate a statistic and a separate pass to use that statistic for the final transformation.
- tl.make_block_ptrtl.make_block_ptr is a Triton API function that creates a descriptor for a block of memory from its base address, shape, strides, offset, block shape, and memory order.
- memory trafficGlobal memory traffic is the movement of data between GPU computation and global memory.
- tile sizeTile size is the configuration specifying the dimensions or element count of a tile processed by a kernel.
- synchronizationSynchronization coordinates concurrent execution so dependent operations observe completed memory writes in the required order.
- numerical stabilityThe property of a numerical computation in which small input or rounding errors do not cause disproportionately large errors in its results.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
