liveBook · Manning

Floating-point operations per second

As explained in Grokking GPU Programming

FLOPS measures the rate of floating-point operations performed per second.

FLOPS measures how many floating-point operations a processor performs each second. Floating-point numbers include decimals such as 3.14; additions and multiplications are examples of operations. FLOPS is a rate, unlike FLOPs, which counts operations. A trillion operations per second is one TFLOPS. section 10.2 section 10.3

FLOPS describes compute capacity, but it does not alone determine kernel speed: data must also reach the processor. For an example GPU rated at 10 TFLOPS with 900 GB/s of memory bandwidth, a kernel reading one float for each operation would need 40 TB/s to keep computation fully supplied—far more than available. section 10.2

Figure 10.1 The GPU memory bottleneck. Computation throughput (10 TFLOPS) is the 8-lane highway on the left, wide enough to move many operations in parallel. Memory bandwidth (900 GB/s) is the single toll gate on the right: every memory access has to pass through it. The arrow marks the bottleneck: no matter how many lanes feed in, the rate at which cars exit is capped by the gate. This imbalance between compute capacity and memory bandwidth is the central problem this chapter addresses.

The highway and toll-gate analogy illustrates why high compute throughput can be limited by memory bandwidth. section 10.2

A rate is not a kernel-speed guarantee

An increment kernel performs one addition while reading and writing a float. Its arithmetic intensity is 0.125 operations per byte; a kernel performing 50 additions per float reaches 6.25. Both remain memory-bound on the example GPU, despite the extra computation. section 10.3

How Grokking GPU Programming covers it

How deep it goes
Defines the rate and distinguishes it from operation counts, then applies it to bandwidth comparisons and the Roofline model.
Examples
Uses an increment kernel, a 50-add kernel, and an example GPU rated at 10 TFLOPS and 900 GB/s.
What it assumes
Defines floating-point numbers with a decimal example and explains operations using additions and multiplications.
What you can do after
Evaluate GPU compute capacity alongside memory bandwidth and recognize when data delivery limits kernel performance.
How it connects
Connects FLOPS to arithmetic intensity, memory bandwidth, the Roofline model, CUDA kernels, and cuBLAS performance.

Also explained in

Compare how 3 books explain floating-point operations per second →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.