chapter one

1 When PyTorch just isn’t enough

 

This chapter covers

  • The CUDA programming model and parallel computing fundamentals
  • Why deep learning operations are suited for GPU parallelization
  • When to use CUDA versus PyTorch
  • The optimization ladder from naive kernels to production performance

Large language models (LLMs) consume staggering compute budgets. The difference between a model that trains in days rather than weeks, or infers in milliseconds rather than seconds, often comes down to how effectively you use GPU hardware. That optimization happens in CUDA, the layer PyTorch calls into when it runs on an NVIDIA GPU.

Writing raw CUDA is the most direct way to control what the GPU executes. This book teaches you to recognize when you need a custom CUDA kernel instead of an existing library and how to write that kernel yourself.

My goal is to take you from curious about GPU programming to a CUDA developer who can spot performance bottlenecks, work close to the hardware, and craft optimization strategies that actually run.

To start, we’ll build your mental model of CUDA from the ground up, beginning with host/device coordination and simple CUDA programs before connecting them to deep learning workloads. We’ll use examples like matrix multiplication to map out parallel opportunities, then point you to the exact tools and first projects you’ll use to run your own CUDA code.

1.1 What is CUDA?

1.2 When do you need custom CUDA?

1.3 CUDA basics

1.3.1 Host and device: Two worlds working together

1.3.2 Kernels: Functions that run on thousands of threads

1.4 Recognizing parallel opportunities

1.4.1 The memory hierarchy: Your performance bottleneck

1.4.2 When parallelism doesn’t help

1.5 Deep learning through the CUDA lens

1.5.1 The nature of deep learning computation

1.5.2 Why GPUs excel at deep learning

1.6 The optimization stack

1.6.1 Starting point: Naive kernels

1.6.2 Optimization layers

Summary