liveBook · Manning

Throughput

As explained in Grokking GPU Programming

Throughput is the amount of work completed per unit of time or over an overall execution period.

Throughput is how much work gets done per unit of time. It contrasts with latency: how long one job takes from start to finish. CPUs prioritize low latency; GPUs are designed for high throughput, completing many simple operations across large amounts of data. section 1.6

That design favors workloads with plenty of independent, similar tasks, such as applying a filter to many pixels or doing matrix math. Having many processing units is not enough by itself: keeping them fed with data matters, and memory bandwidth can be decisive. section 1.6 section 1.10

Table 1.2 Contrasting CPU and GPU design philosophies.
 
CPU (Master Chef)
GPU (Thousand Cooks)
Cores
6–24 on typical desktops
Thousands of simple processing units
Design Goal
Low Latency (Finish each task fast)
High Throughput (Finish many tasks at once)
Analogy
A chef preparing a complex 7-course meal
1,000 workers rolling 1,000 burritos at once
Best For
OS, Web Servers, Complex Logic, and any single-threaded task where one fast core wins
Matrix Math, Image Processing, AI, and many more parallel patterns you’ll discover in this book

Compare CPU and GPU design goals, strengths, and example workloads. section 1.6

How Grokking GPU Programming covers it

How deep it goes
Defines throughput and latency, then explains how GPU cores and memory bandwidth support high throughput.
Examples
Uses image filtering and matrix math as examples of GPU-suited work.
What you can do after
Evaluate workload suitability for GPUs by checking for abundant independent work and regular memory access.
How it connects
Connects throughput with latency, GPUs, CPUs, and memory bandwidth.

Also explained in

Compare how 32 books explain throughput →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.