liveBook · Manning

What you will learn in CUDA for LLMs

Building fast transformer kernels

by Elliot Arledge

Chapter by chapter

Chapter 1: When PyTorch just isn’t enough

This introduction connects effective GPU use with LLM performance and positions raw CUDA as a direct way to control GPU execution.

1.1 What is CUDA? · 1.2 When do you need custom CUDA? · 1.3 CUDA basics · 1.3.1 Host and device: Two worlds working together · 1.3.2 Kernels: Functions that run on thousands of threads · 1.4 Recognizing parallel opportunities · 1.4.1 The memory hierarchy: Your performance bottleneck · 1.4.2 When parallelism doesn’t help · 1.5 Deep learning through the CUDA lens · 1.5.1 The nature of deep learning computation · 1.5.2 Why GPUs excel at deep learning · 1.6 The optimization stack · 1.6.1 Starting point: Naive kernels · 1.6.2 Optimization layers · 1.6.3 The compounding effect · 1.6.4 Scaling to multiple GPUs · 1.6.5 Why this matters · 1.7 Getting ready to build · 1.8 Summary

What you learn in chapter 1 →

Chapter 2: Your first CUDA program

This introduction previews a first complete CUDA program and identifies CPU-side host code as part of the program that launches the vectorAdd kernel.

2.1 The “why”: Throughput vs. latency · 2.2 Your first run · 2.3 Anatomy of a CUDA program: The vecadd.cu file · 2.4 Where the parallelism comes in · 2.5 Scaling up: Grids, blocks, and boundary checks · 2.6 Expanding to 3D arrays · 2.6.1 3D tensor addition: CPU implementation · 2.6.2 The CUDA kernel for 3D tensor addition · 2.6.3 3D kernel launch configuration · 2.6.4 Real-world example: Batch image processing · 2.7 The universal pattern: From nD to 1D · 2.8 Summary

What you learn in chapter 2 →

Chapter 3: Naive kernels

This chapter introduction establishes correctness-first naive CUDA kernels and a CPU-oracle pattern for verifying their outputs.

3.1 From 1D to 3D element-wise addition · 3.1.1 From loops to parallel threads · 3.1.2 Kernel 1, matrix transpose: When data needs to change shape · 3.2 Core neural network kernels · 3.2.1 Kernel 2, GEMM: The heart of neural networks · 3.2.2 Kernel 3, softmax: The final activation · 3.3 Convolution and pooling toolkit · 3.3.1 Kernel 4, 1D convolution: Processing signals and sequences · 3.3.2 Kernel 5, 2D convolution: The engine of computer vision · 3.3.3 Kernel 6, 2D max pooling: Downsampling and feature detection · 3.4 Summary

What you learn in chapter 3 →

Chapter 4: Training and verifying neural nets in raw CUDA

This chapter introduces a progression from a PyTorch reference implementation to a verified and optimized CUDA neural network.

4.1 Level 1: The PyTorch approach · 4.1.1 PyTorch implementation: Setup, training, and benchmarking · 4.2 Level 2: Backpropagation theory · 4.2.1 Building intuition: From slopes to gradients · 4.2.2 Scaling to vectors and matrices · 4.2.3 Layer-by-layer gradient derivations · 4.2.4 Common pitfalls and debugging tips · 4.2.5 Vectorized implementation for batches · 4.3 Level 3: The NumPy approach · 4.3.1 Manual implementation in NumPy · 4.3.2 Verification technique: Isolate and conquer · 4.3.3 Performance · 4.4 Level 4: The C approach · 4.4.1 Setup: Headers, constants, and data structures · 4.4.2 Memory management and initialization · 4.4.3 The forward pass in C · 4.4.4 The backward pass in C · 4.4.5 Training loop and performance · 4.5 Level 5: The naive CUDA approach · 4.5.1 CUDA setup and memory management · 4.5.2 From C loops to CUDA grids: The matmul kernel · 4.5.3 Forward and backward pass kernels · 4.5.4 The training loop and performance · 4.5.5 A note on profiling: Wall-clock vs. GPU time · 4.6 Level 6: The cuBLAS approach · 4.6.1 Integrating cuBLAS and optimizing the training loop · 4.6.2 Dissecting the cublasSgemm API call · 4.6.3 Debugging gradient instability · 4.6.4 Performance and final verification · 4.6.5 A necessary lie and the path forward · 4.7 Summary

What you learn in chapter 4 →

Chapter 5: Transformer inference in CUDA

The chapter introduces transformer inference and contrasts its single-token performance profile with training.

5.1 Getting started: Transformer architecture and setup · 5.1.1 Establishing the baseline scenario · 5.2 Inference: Theory and optimization · 5.2.1 Optimizing autoregressive generation · 5.2.2 Deploying the optimized stack · 5.3 Inference kernels · 5.3.1 Memory layout and contiguity · 5.3.2 GEMV: The inference workhorse · 5.4 Summary

What you learn in chapter 5 →

Chapter 6: Optimizing our transformer kernels

The introduction motivates GPU-aware optimization and previews a profiling-driven approach to improving transformer kernels.

6.1 Preparing to optimize · 6.1.1 Profiling CUDA kernels · 6.2 GEMV · 6.2.1 Understanding the memory access problem · 6.2.2 Achieving coalesced access · 6.2.3 Profiling the coalescing improvement · 6.2.4 Scaling up with block-level reduction · 6.2.5 Maximizing bandwidth with vectorization · 6.2.6 Final GEMV performance analysis · 6.2.7 GEMV optimization summary · 6.3 Softmax · 6.3.1 Online softmax: A better sequential algorithm · 6.3.2 Parallel softmax with warp shuffles · 6.3.3 Fused softmax with shared memory · 6.3.4 Vectorized softmax · 6.3.5 Profiling softmax optimizations · 6.3.6 Softmax optimization summary · 6.4 Layer normalization · 6.4.1 Fused LayerNorm with warp operations · 6.4.2 Fused LayerNorm kernel · 6.4.3 LayerNorm optimization summary · 6.4.4 LayerNorm profiling results · 6.5 Top-k · 6.5.1 A better sequential algorithm: min-heap · 6.5.2 Parallel top-k with block reduction · 6.5.3 Top-k optimization summary · 6.5.4 Top-k profiling analysis · 6.6 General matrix-matrix multiplication · 6.6.1 Understanding the GEMM memory access problem · 6.6.2 Improving coalescing through thread remapping · 6.6.3 GEMM optimization with tiling and shared memory · 6.6.4 1D blocktiling: Computing multiple outputs per thread · 6.6.5 2D blocktiling: Maximum register reuse · 6.6.6 Vectorizing memory access for tiling · 6.6.7 GEMM profiling: Understanding the bottleneck · 6.6.8 GEMM optimization summary · 6.7 Summary

What you learn in chapter 6 →

Chapter 7: Tensor cores

Introduces tensor cores as specialized hardware for accelerating the matrix multiplications central to deep learning workloads.

7.1 Tensor core fundamentals · 7.1.1 What are tensor cores? · 7.1.2 When to use tensor cores · 7.1.3 A brief history of tensor core instructions · 7.1.4 Tensor core performance across GPU generations · 7.2 WMMA: The high-level tensor core API · 7.2.1 WMMA fragments and tiling strategy · 7.2.2 Loading data with WMMA · 7.2.3 Matrix multiply with WMMA · 7.2.4 Storing results · 7.3 WGMMA: Asynchronous warp group operations · 7.3.1 Basic WGMMA: The first asynchronous tensor cores · 7.3.2 Understanding PTX inline assembly · 7.3.3 WGMMA with larger tiles · 7.3.4 WGMMA with asynchronous memory operations · 7.3.5 WGMMA with maximum tiles · 7.4 Performance analysis: CUDA cores vs. tensor cores · 7.5 Performance progression analysis · 7.5.1 Decision framework: When to use tensor cores · 7.6 From CUDA cores to tensor cores: Pattern mapping · 7.6.1 Manual tiling becomes hardware tiling · 7.6.2 Vectorized loads become TMA · 7.6.3 Shared memory becomes implicit · 7.6.4 What you still optimize manually · 7.7 TCGen05: Blackwell’s tensor cores · 7.7.1 TCGen05 overview · 7.7.2 Challenges and considerations · 7.8 Looking ahead · 7.9 Summary

What you learn in chapter 7 →

Chapter 8: Flash Attention

This introduction contrasts tensor-core compute throughput with attention’s data-movement bottleneck and previews Flash Attention.

8.1 From compute-bound to memory-bound · 8.1.1 The HBM bottleneck in standard attention · 8.2 Designing Flash Attention · 8.2.1 Building intuition: Why this problem is harder than it looks · 8.2.2 The Flash Attention solution: Tiling and online softmax · 8.2.3 Tiling: Breaking the problem down · 8.2.4 Online softmax: Computing global statistics iteratively · 8.3 Implementing Flash Attention · 8.3.1 Dissecting the naive implementation · 8.3.2 Understanding the work distribution strategy · 8.3.3 The three-level loop hierarchy · 8.3.4 The fused Flash Attention kernel · 8.3.5 Benchmarking our kernel · 8.4 The evolution of Flash Attention · 8.4.1 Flash Attention 1 and 2 · 8.4.2 Flash Attention 3: Asynchrony and low precision · 8.4.3 Flash Attention 4: Complexity for Blackwell · 8.4.4 What this evolution teaches us · 8.5 Summary

What you learn in chapter 8 →

Chapter 9: Quantization

The chapter introduction explains how quantization can make large models fit on smaller GPUs and trade some accuracy for faster, lower-memory inference.

9.1 Quantization building blocks · 9.1.1 Data types: The foundation of quantization · 9.1.2 Basic quant/dequant operations · 9.2 Deployment strategies · 9.2.1 Granularity schemes: How to apply quantization · 9.2.2 Calibration: Computing quantization parameters · 9.2.3 Static vs. dynamic quantization · 9.2.4 Symmetric vs. asymmetric quantization · 9.3 Advanced techniques · 9.3.1 Activation-aware weight quantization · 9.3.2 Simple example: Big vs. small inputs · 9.4 Putting it all together · 9.4.1 Real model quantization workflow · 9.4.2 Accuracy validation strategy · 9.4.3 Common pitfalls · 9.4.4 The tradeoffs in one place · 9.5 Where we can go further · 9.5.1 GPT quantization · 9.5.2 NormalFloat4 · 9.5.3 Quantization-aware training · 9.5.4 Quantization kernel profiling · 9.5.5 The road ahead · 9.6 Exercises · 9.7 Summary

What you learn in chapter 9 →

Chapter 10: Distributed computing

This introduction explains how PCIe and NVLink interconnects affect communication performance within a multi-GPU server.

10.1 Understanding the hardware · 10.1.1 Lab 1: Mapping your hardware · 10.1.2 Understanding NCCL’s primitives · 10.1.3 Ring vs. tree algorithms · 10.2 Tensor parallelism · 10.2.1 Initialization and GPU assignment · 10.2.2 Memory allocation · 10.2.3 Local matrix multiplication · 10.2.4 Combining the results with AllReduce · 10.3 Pipeline parallelism · 10.3.1 Naive pipeline parallelism implementation: Sequential processing · 10.3.2 Optimized implementation: CUDA streams · 10.3.3 Why streams work · 10.4 Evaluating scaling strategies · 10.4.1 Parallelism strategies · 10.4.2 When to use each strategy · 10.5 Summary

What you learn in chapter 10 →

Chapter 11: CUTLASS

The introduction positions CUTLASS as a way to gain more control over GEMM performance than a black-box library provides.

11.1 Why CUTLASS matters · 11.1.1 The core idea: Decomposing a GEMM · 11.2 Building our first CUTLASS kernel · 11.2.1 Defining the kernel’s shape · 11.2.2 Composing the kernel with CollectiveBuilder · 11.2.3 Launching the kernel from the host · 11.3 CuTe: The language of layouts · 11.4 Tuning for performance · 11.5 Architecture and precision · 11.5.1 Across the generations: Ampere vs. Hopper · 11.5.2 Narrow precision operations: FP4 on Blackwell · 11.6 Profiling and verification · 11.7 Beyond a single GPU: Distributed GEMM · 11.8 Summary

What you learn in chapter 11 →

Chapter 12: Now what?

This introduction explains the book’s deliberately narrow focus on durable CUDA fundamentals and frames the final chapter as a guide to continued learning.

12.1 What this book intentionally left out · 12.1.1 Data center–level considerations · 12.1.2 Convolution-heavy architectures · 12.1.3 The Blackwell TCGen05 instructions · 12.2 Choosing the right abstraction layer · 12.2.1 Triton: Python-level GPU programming · 12.2.2 torch.compile · 12.3 Serving systems and end-to-end performance · 12.3.1 vLLM · 12.3.2 SGLang · 12.3.3 TensorRT-LLM · 12.3.4 ONNX · 12.4 Compiler and generated-kernel workflows · 12.5 High-level frameworks and languages · 12.5.1 JAX · 12.5.2 Mojo · 12.6 CUTLASS and template libraries · 12.6.1 Python CuTe DSL · 12.6.2 CUTLASS profiler and auto-tuning · 12.7 Expanding your toolkit · 12.7.1 GPU MODE · 12.7.2 Ultra-Scale Playbook · 12.7.3 LeetGPU · 12.8 High-level inference tricks · 12.9 Running on edge hardware · 12.10 Charting your next moves · 12.10.1 The meta-point: The fundamentals generalize · 12.10.2 Where to go from here · 12.10.3 Final thoughts · 12.10.4 Resources · 12.11 Summary

What you learn in chapter 12 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.