Chapter 1: Introduction to Triton
The introduction explains why growing AI compute demands make GPUs important, why Triton fills a gap between ML frameworks and CUDA, and how combining operations in one kernel can reduce overhead.
After this chapter you will be able to
- Triton’s programming tradeoff. Explain the gap between ML frameworks and CUDA that Triton addresses, and describe the tradeoff Triton offers.
- Kernel fusion overhead. Describe a GPU kernel and explain why combining operations in one kernel can avoid costs of PyTorch eager execution.
- Triton’s Python-first tradeoff. Explain how Triton offers GPU-level optimization from Python without the workflow disruption of rewriting a prototype in CUDA. Describe the section’s stated division of labor: the programmer expresses the optimization, while Triton’s compiler generates optimized GPU code.
- Python language and compiler. Explain that Triton combines a Python-embedded DSL with a compiler that turns functions decorated with @triton.jit into GPU code when first called. Distinguish this workflow from writing and compiling CUDA C++ kernels ahead of time.
- Tile based programming model. Explain how writing operations on whole tensor tiles differs from writing code for individual scalar threads, and describe the compiler's role in mapping tile operations to hardware.
- Backend portability and runtime. Explain why the same tile-based kernel source can target different GPU vendors, while still requiring the appropriate runtime for the selected hardware.
- Triton and CUDA tradeoff. Explain why Triton is often a productive choice for custom GPU kernels and when CUDA may be preferable.
- Automatic optimization and custom kernels. Distinguish what torch.compile() does automatically from the situation in which writing a Triton kernel yourself becomes useful.
- Compute APIs for GPU work. Explain that the book uses GPU programming to mean general-purpose computation rather than graphics rendering, and that a compute API lets a host program express and launch work on the GPU.
- GPU abstraction trade-off. Explain how high-level libraries and low-level APIs differ in productivity, control, and the kinds of GPU operations they make easy to implement.
- Triton middle ground. Explain where Triton fits between high-level libraries and CUDA, and when its approach is useful.
- Reasons for multiple GPU DSLs. Explain why no single GPU DSL serves every kernel and GPU generation equally well, and why languages differ in which decisions they leave to the programmer or compiler.
- Helion Triton Gluon control tradeoff. Distinguish what the programmer and compiler decide in Helion, Triton, and Gluon, including the benefit and cost of the extra control Gluon provides.
- Triton in PyTorch compilation. Explain how a PyTorch user can run Triton-generated code without writing a Triton kernel.
- Triton in LLM training. Describe how Unsloth and Liger-Kernel use Triton in LLM training.
- Portable inference attention backends. Explain why vLLM and SGLang use Triton attention backends across GPU vendors.
- GPU hardware and performance. Explain why understanding the GPU matters for Triton-powered libraries seeking high performance.
- Concurrency and parallelism distinction. Distinguish overlapping progress through task switching from tasks executing at the same instant on multiple workers.
- Independent tasks enable parallelism. Explain why dividing code into independent tasks lets GPU hardware execute useful work in parallel.
- Waiting versus arithmetic workloads. Explain why concurrency alone can help waiting-heavy work, whereas the tensor workloads discussed here depend on parallelism for speed.
- Independent data parallel work. Explain why element-wise operations on large datasets suit a GPU: each output can be computed independently, allowing many tasks to run at once.
- CPU and GPU design tradeoff. Distinguish the CPU's latency-optimized design from the GPU's throughput-optimized design, including how each allocates chip resources and what that means for individual threads versus total work completed.
- Host device responsibilities and data. Trace the CPU host's and GPU device's roles in launching GPU work, and describe where a PyTorch result resides after a GPU operation and how it moves back to CPU memory.
- Triton tile-level programming. Distinguish Triton’s block-level perspective from CUDA’s thread-level perspective, including what the programmer specifies and what Triton manages.
- GPU execution hierarchy. Describe how threads form warps, warps form blocks, and blocks make up a grid, distinguishing the logical grid from the hardware SMs that execute its blocks.
- Grid sizing and block scheduling. Apply tile size to determine a grid size, and trace how SMs execute its blocks until the grid is complete.
- Data bottlenecks and latency hiding. Explain why a GPU can wait despite having many arithmetic units, and how running many threads keeps it busy during slow memory accesses.
- Memory hierarchy and storage roles. Describe the hierarchy’s size-and-speed tradeoff and distinguish what registers, shared memory, caches, and global memory do.
- Triton kernel execution path. Trace a Triton kernel from its Python definition through first-call compilation, later reuse, launch, and scheduling.
Section by section
Introduction
The introduction explains why growing AI compute demands make GPUs important, why Triton fills a gap between ML frameworks and CUDA, and how combining operations in one kernel can reduce overhead.
- GPUs in AI scaling: Explain why GPUs have become central to modern AI workloads as training data and compute demands have grown.
Why it matters: It explains why gaining more control over GPU computation is worth learning. - Triton’s programming tradeoff: Explain the gap between ML frameworks and CUDA that Triton addresses, and describe the tradeoff Triton offers.
Why it matters: This is the motivation for using Triton when a framework’s ready-made operations are not enough. - Kernel fusion overhead: Describe a GPU kernel and explain why combining operations in one kernel can avoid costs of PyTorch eager execution.
Why it matters: It gives a concrete reason a custom kernel can be faster than a sequence of separate framework operations.
Key terms: GPU, compute demands, compute budgets, Triton, GPU kernels, CUDA, low-level performance, GPU kernel, eager execution, kernel launch
1.1 The philosophy and foundations of Triton
This section explains why a correct PyTorch prototype may still need GPU-level optimization and how Triton makes that optimization accessible from Python.
- GPU underutilization in prototypes: Explain how profiling can show that a correct PyTorch prototype leaves the GPU underutilized because time is spent launching many small kernels and moving data between the CPU and GPU rather than computing.
Why it matters: It explains why correctness alone may not demonstrate an algorithm’s performance potential. - Triton’s Python-first tradeoff: Explain how Triton offers GPU-level optimization from Python without the workflow disruption of rewriting a prototype in CUDA. Describe the section’s stated division of labor: the programmer expresses the optimization, while Triton’s compiler generates optimized GPU code.
Why it matters: It captures the reason Triton is presented as an alternative to both a high-level prototype and a CUDA rewrite.
Key terms: profiler, per-operation breakdown, GPU underutilized, kernels, Triton, Python-first, GPU kernels, compiler, CUDA
1.1.1 What is Triton?
This section defines Triton as a Python-embedded GPU language and compiler, then explains why its specialized, tile-based model supports productivity, performance, and portability.
- Python language and compiler: Explain that Triton combines a Python-embedded DSL with a compiler that turns functions decorated with @triton.jit into GPU code when first called. Distinguish this workflow from writing and compiling CUDA C++ kernels ahead of time.
Why it matters: This explains how Triton keeps GPU kernel development in a Python workflow. - Specialized embedded language scope: Describe what makes Triton an embedded DSL and why its focus on blocked tensor operations can combine Python-level productivity with performance close to hand-tuned CUDA.
Why it matters: Triton's restricted scope clarifies both what it is useful for and why its Python interface need not mean low performance. - Tile based programming model: Explain how writing operations on whole tensor tiles differs from writing code for individual scalar threads, and describe the compiler's role in mapping tile operations to hardware.
Why it matters: Thinking in tiles is the shift behind Triton's approach, not merely a convenience of Python syntax. - Backend portability and runtime: Explain why the same tile-based kernel source can target different GPU vendors, while still requiring the appropriate runtime for the selected hardware.
Why it matters: Portable kernel source does not eliminate the need for a working GPU software stack on each machine.
Key terms: embedded domain-specific language, compiler, @triton.jit, runtime, domain-specific language, embedded DSL, blocked tensor operations, block-based language, tile, block
1.1.2 When to use Triton
This section explains when writing a Triton kernel is worthwhile, compared with relying on torch.compile() or choosing lower-level GPU programming.
- Triton and CUDA tradeoff: Explain why Triton is often a productive choice for custom GPU kernels and when CUDA may be preferable.
Why it matters: The right tool depends on how much control you need relative to development effort. - Automatic optimization and custom kernels: Distinguish what torch.compile() does automatically from the situation in which writing a Triton kernel yourself becomes useful.
Why it matters: You need not write a kernel when automatic optimization suffices, but you have an option for a specialized bottleneck it does not resolve.
Key terms: Triton, CUDA, hand-tuned CUDA, PyTorch pipeline, torch.compile(), TorchInductor, Triton kernels, custom activation, unusual normalization layer
1.2 How Triton fits into the GPU ecosystem
Defines GPU programming in this book as using compute APIs and briefly situates Triton among other interfaces in a CUDA-dominated ecosystem.
- Compute APIs for GPU work: Explain that the book uses GPU programming to mean general-purpose computation rather than graphics rendering, and that a compute API lets a host program express and launch work on the GPU.
Why it matters: This establishes what the book means by GPU programming before comparing programming approaches. - CUDA dominance and alternatives: Describe CUDA as the dominant interface despite the existence of other compute APIs and Python-native eDSLs, and explain how NVIDIA’s toolchain investment and mutually reinforcing hardware and developer adoption contributed to that dominance.
Why it matters: It explains why CUDA is prominent without treating it as the only way to program GPUs.
Key terms: compute APIs, general-purpose computation, host program, CUDA, Python-native eDSLs, toolchain investment
1.2.1 Levels of abstraction in GPU programming
This section places Triton on a GPU programming abstraction spectrum and explains the trade-off between ease of use and control over execution.
- GPU abstraction trade-off: Explain how high-level libraries and low-level APIs differ in productivity, control, and the kinds of GPU operations they make easy to implement.
Why it matters: The trade-off explains why a tool between a framework and a low-level API is useful. - PTX and SASS levels: Distinguish PTX from SASS and explain why writing at the instruction level is uncommon.
Why it matters: It clarifies what lies below CUDA on the abstraction spectrum without implying that most developers should program there. - Triton middle ground: Explain where Triton fits between high-level libraries and CUDA, and when its approach is useful.
Why it matters: This is the section's reason for choosing Triton for custom GPU operations.
Key terms: spectrum of abstraction, control, productivity, high-level libraries, CUDA, PTX, ptxas, SASS, virtual assembly language, Triton
1.2.2 The Python GPU DSL landscape
This section explains why Python GPU DSLs make different tradeoffs between programmer control and compiler automation, placing Helion, Triton, and Gluon along that spectrum.
- Reasons for multiple GPU DSLs: Explain why no single GPU DSL serves every kernel and GPU generation equally well, and why languages differ in which decisions they leave to the programmer or compiler.
Why it matters: This explains why choosing a GPU language is also choosing how much hardware control to take on. - Helion Triton Gluon control tradeoff: Distinguish what the programmer and compiler decide in Helion, Triton, and Gluon, including the benefit and cost of the extra control Gluon provides.
Why it matters: The comparison helps you recognize when easier tuning or more direct hardware control is the relevant tradeoff.
Key terms: Python DSLs, memory layout, hardware scheduling, compiler, Helion, Triton, Gluon, autotuner, blocked kernels, tile shapes
1.2.3 Triton in the wild: libraries and use cases
The section shows how Triton is used in mainstream ML frameworks, LLM training libraries, and inference engines, and distinguishes it from NVIDIA’s similarly named inference server.
- Triton in PyTorch compilation: Explain how a PyTorch user can run Triton-generated code without writing a Triton kernel.
Why it matters: Triton can affect your PyTorch workloads even when you do not program in Triton directly. - Triton in LLM training: Describe how Unsloth and Liger-Kernel use Triton in LLM training.
Why it matters: These examples show why libraries adopt Triton for performance-critical training work. - Portable inference attention backends: Explain why vLLM and SGLang use Triton attention backends across GPU vendors.
Why it matters: A shared kernel source can reduce the work needed to support different accelerators. - Two unrelated Triton projects: Distinguish the Triton programming language from NVIDIA Triton Inference Server.
Why it matters: The distinction prevents confusion when looking up Triton resources for GPU programming.
Key terms: PyTorch 2.x, torch.compile, TorchInductor, Triton kernels, Unsloth, Liger-Kernel, attention, backpropagation, vLLM, SGLang
1.3 Thinking in parallel : GPU execution model
The section explains why understanding GPU hardware matters before writing performance-focused Triton code.
- GPU hardware and performance: Explain why understanding the GPU matters for Triton-powered libraries seeking high performance.
Why it matters: It connects Triton performance work to the machine running the code.
Key terms: Triton-powered libraries, GPU, underlying hardware
1.3.1 Concurrency vs parallelism
The section distinguishes concurrency from parallelism, explains why tensor workloads need parallel execution, and clarifies a basic difference in GPU terminology across vendors.
- Concurrency and parallelism distinction: Distinguish overlapping progress through task switching from tasks executing at the same instant on multiple workers.
Why it matters: The distinction prevents you from mistaking progress on several tasks for simultaneous computation. - Independent tasks enable parallelism: Explain why dividing code into independent tasks lets GPU hardware execute useful work in parallel.
Why it matters: Splitting work into independent pieces is what gives GPU processors work they can do simultaneously. - Waiting versus arithmetic workloads: Explain why concurrency alone can help waiting-heavy work, whereas the tensor workloads discussed here depend on parallelism for speed.
Why it matters: It explains why GPU tensor performance calls for simultaneous computation rather than merely juggling tasks. - Vendor names for GPU units: State that GPU execution-unit names vary by vendor, including NVIDIA's Streaming Multiprocessors and AMD's Compute Units.
Why it matters: Recognizing vendor-specific names makes GPU descriptions easier to compare.
Key terms: concurrency, parallelism, independent tasks, waiting, tensor operations, arithmetic, Streaming Multiprocessors, SMs, Compute Units, CUs
1.3.2 Why GPUs are designed differently
The section explains why GPU hardware favors large amounts of independent work, when CPU execution can still be preferable, and how the CPU and GPU cooperate in a GPU-accelerated program.
- Independent data parallel work: Explain why element-wise operations on large datasets suit a GPU: each output can be computed independently, allowing many tasks to run at once.
Why it matters: Independence helps explain which work can benefit from massive GPU parallelism. - CPU and GPU design tradeoff: Distinguish the CPU's latency-optimized design from the GPU's throughput-optimized design, including how each allocates chip resources and what that means for individual threads versus total work completed.
Why it matters: The hardware tradeoff explains why a GPU can win on total throughput without making each task faster. - Workload dependent processor choice: Explain why a CPU can be faster for some work despite GPU parallelism, considering workload size, branching, sequential dependencies, and the costs of transferring data and launching a kernel.
Why it matters: Choosing a processor depends on the whole workload, not just the GPU's peak parallelism. - Host device responsibilities and data: Trace the CPU host's and GPU device's roles in launching GPU work, and describe where a PyTorch result resides after a GPU operation and how it moves back to CPU memory.
Why it matters: Keeping track of who launches the work and where tensors live is a foundation for writing efficient Triton programs.
Key terms: embarrassingly parallel, element-wise tensor operations, data-parallel, latency-optimized, throughput-optimized, caches, control logic, ALUs, kernel launch, SIMD
1.3.3 The GPU execution model
This section explains how Triton’s tile-level kernels map onto GPU threads, warps, blocks, grids, and streaming multiprocessors during execution.
- Triton tile-level programming: Distinguish Triton’s block-level perspective from CUDA’s thread-level perspective, including what the programmer specifies and what Triton manages.
Why it matters: It identifies the unit of work you actually describe when writing a Triton kernel. - GPU execution hierarchy: Describe how threads form warps, warps form blocks, and blocks make up a grid, distinguishing the logical grid from the hardware SMs that execute its blocks.
Why it matters: This hierarchy connects a Triton launch to the hardware that carries out its work. - Warp instruction behavior: Explain how GPU threads in a warp normally execute instructions and why this differs from the independent CPU-thread picture.
Why it matters: It explains the parallel behavior beneath Triton’s tile-level description without treating GPU threads like independent CPU threads. - Grid sizing and block scheduling: Apply tile size to determine a grid size, and trace how SMs execute its blocks until the grid is complete.
Why it matters: It connects the number of tiles you launch to how the GPU completes a workload larger than it can run at once.
Key terms: program instance, block, tile, kernel, grid, thread, warp, streaming multiprocessor, GPU thread, CPU thread
1.4 How GPUs handle data
Explains why moving data can limit GPU performance, how GPUs hide memory latency and organize storage, and how a Triton kernel reaches execution.
- Data bottlenecks and latency hiding: Explain why a GPU can wait despite having many arithmetic units, and how running many threads keeps it busy during slow memory accesses.
Why it matters: It explains why abundant GPU compute does not automatically make a kernel fast. - Memory hierarchy and storage roles: Describe the hierarchy’s size-and-speed tradeoff and distinguish what registers, shared memory, caches, and global memory do.
Why it matters: Where data sits when it is needed can strongly affect kernel speed. - SRAM and DRAM tradeoff: Distinguish why the GPU uses SRAM for nearby storage and DRAM for global memory.
Why it matters: The underlying technologies help explain the hierarchy’s speed-and-capacity tradeoff. - Triton kernel execution path: Trace a Triton kernel from its Python definition through first-call compilation, later reuse, launch, and scheduling.
Why it matters: It helps separate first-call compilation and launch overhead from time spent doing GPU work.
Key terms: distance problem, latency hiding, clock cycles, registers, shared memory, L1 cache, L2 cache, global memory, memory hierarchy, SRAM
1.5 Summary
1.6 Exercises
Where this is used later
- Kernel fusion: the solution (chapter 3) · Expands on combining operations through kernel fusion.
- A high-level language with a low-level compiler (chapter 2) · Develops the relationship between Triton’s high-level language and low-level compiler.
- Imports and the JIT decorator (chapter 2) · Introduces the JIT decorator in a complete kernel.
- The Triton programming model (chapter 2) · Introduces Triton as a GPU programming tool.
- Thinking in blocks, not threads (chapter 2) · Develops the distinction between programming blocks and programming threads.
- A complete Triton kernel: vector addition end to end (chapter 2) · Shows how to write a Triton kernel rather than relying solely on automatically generated code.
- Embarrassingly parallel tasks: the baseline GPU pattern (chapter 3) · Introduces embarrassingly parallel tasks as a baseline GPU pattern.
- The launch grid (chapter 2) · Builds on the distinction between the CPU-side launch and GPU-side kernel execution.
Concepts explained in this chapter
- GPU kernelA GPU kernel is a function launched on a GPU to perform computation in parallel.
- TritonThe Triton compiler transforms Triton programs into optimized GPU code for a target backend.
- GPUA GPU device is the complete graphics-processing hardware containing the streaming multiprocessors and their connected memory system.
- CUDACUDA is NVIDIA's parallel computing platform and programming model, including the CUDA C++ language extensions for GPU programming.
- @triton.jit@triton.jit is a Triton decorator that marks a Python function for JIT compilation into a GPU kernel.
- streaming multiprocessorA streaming multiprocessor is NVIDIA's basic GPU computational building block that executes groups of threads and provides computational resources.
- thread blockA GPU block is a group of threads scheduled as a unit that can cooperate through shared memory.
- threadA CPU thread is an independently schedulable sequence of instructions executed by a CPU.
- gridA GPU grid is the complete collection of blocks or program instances launched for one kernel call.
- warpA warp is a hardware group of threads that an NVIDIA streaming multiprocessor schedules and issues together.
- block-level programmingA block-level programming language lets a programmer express a GPU kernel from the perspective of one block processing one tile.
- GPU memoryVideo memory is memory dedicated to storing data for access by a graphics processing unit.
- L1 cacheAn L1 cache is a small, hardware-managed cache close to the GPU compute units.
- L2 cacheAn L2 cache is a hardware-managed GPU cache that is generally larger and farther from compute units than the L1 cache.
- hardware registerA register is a small, private storage location associated with a GPU thread for operands, intermediate results, and other active values.
- memory hierarchyA memory hierarchy is an ordered arrangement of memory levels that trade capacity and access speed to keep frequently needed data near computation.
- shared memoryShared memory is explicitly managed on-chip GPU memory visible to all threads in a single block.
- SIMTThe Single Instruction, Multiple Threads (SIMT) execution model runs the same instruction across multiple threads operating on separate data.
- program instanceA program instance is Triton's unit of work corresponding to one block of input elements in a compiled kernel launch.
- tileA tile is a portion of a tensor processed by one Triton program instance.
- DRAMDRAM is a capacitor-based memory technology that requires periodic refresh and provides high storage density at higher access latency than SRAM.
- SRAMSRAM is a transistor-based memory technology that retains state while powered and provides low-latency access without periodic refresh.
- clock cycleA clock cycle is the interval between consecutive pulses of a processor's internal clock.
- distance problemThe performance problem caused by the latency of moving data from storage to computational units.
- latency hidingA parallel execution technique that keeps processing units busy by running other threads while some threads wait for memory operations.
- CPUA central processing unit is a general-purpose processor optimized for low-latency sequential and moderately parallel execution.
- data parallelismData parallelism applies the same computation independently to multiple partitions of data.
- deviceIn CUDA terminology, the device is the GPU-side processor and execution environment that executes dispatched parallel work.
- hostIn GPU computing, the host is the CPU-side execution environment that launches and orchestrates device work.
- latency optimizationA latency-optimized design minimizes the time required to complete an individual task or thread of execution.
- throughput-optimized designA throughput-optimized design maximizes the amount of work completed over a given period, often by executing many tasks concurrently.
- domain-specific languageA domain-specific language is a programming language designed for a particular class of problems.
- concurrencyConcurrency is the organization of multiple tasks so that their progress overlaps, often by interleaving work rather than executing tasks simultaneously.
- parallelismParallelism is the simultaneous execution of multiple tasks by multiple physical or computational resources.
- autotuningAutotuning is the process of evaluating alternative implementation configurations and selecting one based on measured performance.
- element-wise operationAn element-wise tensor operation applies the same function independently to individual tensor elements.
- embarrassingly parallelAn embarrassingly parallel task is a computation whose output elements depend independently on separate inputs and require no coordination between work instances.
- GPU runtimeA GPU runtime is the software layer that provides device memory management, kernel dispatch, and execution support for a GPU backend.
- ROCmROCm is AMD's open-source GPU computing stack comprising driver, runtime, and compiler-toolchain components.
- block-based languageA block-based language expresses computation over whole data blocks rather than individual scalar threads.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
