liveBook · Manning

GPU

As explained in GPU Programming with Triton

A GPU device is the complete graphics-processing hardware containing the streaming multiprocessors and their connected memory system.

A GPU is a device that combines many processing units with a memory system to execute large amounts of work in parallel. It is designed for throughput: completing many tasks over time, rather than minimizing the latency of one task. Element-wise tensor operations are a natural fit because each output can be computed independently. section 1.3.2

The CPU, or host, orchestrates work; the GPU, or device, executes it. A compute API lets the host launch general-purpose work on the device. Triton provides a Python-based way to describe kernels, while its compiler handles decisions such as memory coalescing and instruction scheduling. section 1.2 section 1.5 section 2.1

A kernel launch forms a grid of blocks. Each block is assigned to a streaming multiprocessor (SM), which may run several blocks at once if resources permit. As blocks finish, SMs take more from the grid. On NVIDIA GPUs, threads within a block are scheduled in groups of 32 called warps. section 1.3.3

Figure 1.6 The relationship between the software execution model (threads, blocks, and grids) and the underlying GPU hardware (cores, streaming multiprocessors, and device). Each level of the software model corresponds to a specific layer of GPU hardware responsible for executing it..
Growth in training data

Map the software hierarchy of threads, blocks, and grids to GPU hardware. section 1.3.3

Scheduling a grid on an H100

A block of 256 threads forms eight NVIDIA warps. For a grid of 1,024 such blocks on an H100 SXM with 132 SMs, assuming four blocks can reside on each SM, 528 blocks can run concurrently. The remaining blocks are scheduled as SMs finish work. Actual residency depends on resources such as registers and shared memory. section 1.3.3

Compute capacity alone does not guarantee speed: arithmetic units can wait hundreds of clock cycles for data from high-bandwidth memory. GPUs help hide that latency by running other ready threads while some wait. Memory placement and data movement therefore matter as much as parallel arithmetic. section 1.4

Figure 1.8 The memory hierarchy organized by speed and size. Registers top the hierarchy as the fastest and smallest storage; global memory sits at the bottom as the largest and slowest. Shared memory and the L1 cache appear on adjacent rows to reflect their distinct programming roles; on NVIDIA GPUs from the Volta architecture onwards, they share the same physical on-chip SRAM on each SM and have the same access latency, so the figure separates them by function, not by speed.
CH01 F07 Fartale

Compare the GPU memory hierarchy by size, speed, and programming role. section 1.4

When assessing a GPU for kernel performance, identify:

  • Its model, SM or compute-unit count, and architecture family. section 3.6.1
  • Its memory type, capacity, peak bandwidth, and whether memory is integrated or dedicated. section 3.6.1
  • Its clock frequencies and supported compute APIs. section 3.6.1

How GPU Programming with Triton covers it

How deep it goes
Explains throughput design, grid scheduling, and the memory hierarchy, then provides a hardware-inspection checklist.
Examples
Uses element-wise tensor operations and an H100 grid with 256-thread blocks to illustrate parallel work and scheduling.
What you can do after
Evaluate a GPU's compute resources and memory characteristics when reasoning about kernel performance.
How it connects
Connects GPUs to data parallelism, streaming multiprocessors, memory hierarchy, and Triton kernels.

Also explained in

Compare how 19 books explain GPU →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.