liveBook · Manning

CUDA

As explained in GPU Programming with Triton

CUDA is NVIDIA's parallel computing platform and programming model, including the CUDA C++ language extensions for GPU programming.

CUDA is NVIDIA’s parallel computing platform and programming model for general-purpose GPU work. It lets a host program launch computations on a GPU, including workloads in deep learning and scientific computing. CUDA C++ provides a low-level way to write these GPU kernels. section 1.2 section 1.1.1

A CUDA kernel runs work in parallel on the device after being launched by the host. Threads are the basic units of execution: each can process its own piece of data. Threads form warps; on NVIDIA GPUs, a warp contains 32 threads. Blocks group work within a grid, and the GPU schedules blocks across streaming multiprocessors. section 1.3.3 section 1.5

Figure 1.6 The relationship between the software execution model (threads, blocks, and grids) and the underlying GPU hardware (cores, streaming multiprocessors, and device). Each level of the software model corresponds to a specific layer of GPU hardware responsible for executing it..
Growth in training data

The execution hierarchy connects threads, blocks, and grids to the GPU hardware that runs them. section 1.3.3

Low-level CUDA programming gives developers control over how work is divided into threads and blocks, how data moves between global and shared memory, and how threads synchronize. That control can deliver excellent performance, but indexing, memory alignment, and parallel coordination add complexity. section 1.2.1

Matrix multiplication

A framework call such as torch.matmul(A, B) can provide efficient GPU execution without exposing its implementation. A CUDA implementation offers more control over execution, but even this operation can require hundreds of lines to manage its low-level details. section 1.2.1

Triton sits between high-level frameworks and CUDA: it lets developers write GPU kernels in Python while its compiler manages much of the thread-level work. A Triton program instance corresponds to what CUDA calls a block. On NVIDIA hardware, Triton requires CUDA 11.8 or later and compiles kernels to PTX; NVIDIA’s ptxas then lowers PTX to GPU-specific machine code. section 1.3.3 section 1.1.1 section 1.2.1

How GPU Programming with Triton covers it

How deep it goes
Explains execution terminology and control over memory, work division, and synchronization, without walking through a CUDA kernel.
Examples
Contrasts torch.matmul with a CUDA matrix multiplication implementation and uses tensor elements and pixels to illustrate thread work.
What it assumes
Assumes little or no CUDA experience and introduces GPU execution terminology for Triton users.
What you can do after
Enables evaluation of CUDA’s control and complexity against framework and Triton alternatives.
How it connects
Connects CUDA to GPU kernels, threads, blocks, warps, shared memory, Triton, PTX, and SASS.

Also explained in

Compare how 7 books explain CUDA →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.