1 When PyTorch just isn’t enough
This chapter covers
Large language models (LLMs) consume staggering compute budgets. The difference between a model that trains in days rather than weeks, or infers in milliseconds rather than seconds, often comes down to how effectively you use GPU hardware. That optimization happens in CUDA, the layer PyTorch calls into when it runs on an NVIDIA GPU.
Writing raw CUDA is the most direct way to control what the GPU executes. This book teaches you to recognize when you need a custom CUDA kernel instead of an existing library and how to write that kernel yourself.
NOTE Higher-level abstractions like OpenAI’s Triton exist, but understanding raw CUDA gives you deeper control and insight.
My goal is to take you from curious about GPU programming to a CUDA developer who can spot performance bottlenecks, work close to the hardware, and craft optimization strategies that actually run.
To start, we’ll build your mental model of CUDA from the ground up, beginning with host/device coordination and simple CUDA programs before connecting them to deep learning workloads. We’ll use examples like matrix multiplication to map out parallel opportunities, then point you to the exact tools and first projects you’ll use to run your own CUDA code.