Floating-point operations per second
Floating-point operations per second (FLOP/s) is a measure of computational throughput expressed as the number of floating-point operations performed per second.
Grokking GPU Programming
FLOPS measures the rate of floating-point operations performed per second.
- How it is used
- Compares GPU compute capacity with memory bandwidth to explain performance limits and classify kernels.
- How deep it goes
- Defines the rate and distinguishes it from operation counts, then applies it to bandwidth comparisons and the Roofline model.
- Examples
- Uses an increment kernel, a 50-add kernel, and an example GPU rated at 10 TFLOPS and 900 GB/s.
- What you can do after
- Evaluate GPU compute capacity alongside memory bandwidth and recognize when data delivery limits kernel performance.
Read how Grokking GPU Programming explains floating-point operations per second →
GPU Programming with Triton
Floating-point operations per second measures the rate at which a processor or system performs floating-point arithmetic.
- How it is used
- Measures floating-point throughput as a performance metric for processors and computing systems.
- How deep it goes
- Defines individual operations, total operation counts, and rates, then works through a simple timing calculation.
- Examples
- Uses a 1,000-add loop and a GPU rated at 10 TFLOPS to illustrate measured and peak throughput.
- What you can do after
- Evaluate throughput by dividing a floating-point operation count by elapsed time.
Read how GPU Programming with Triton explains floating-point operations per second →
CUDA for LLMs
TFLOPS measures computational throughput in trillions of floating-point operations per second.
- How it is used
- Uses TFLOPS to assess compute-bound GEMM performance alongside compute-unit throughput.
- How deep it goes
- Interprets measured TFLOPS through a kernel-optimization ladder, from naive access patterns to vectorized tiling.
- Examples
- Compares naive and vectorized 4096 × 4096 FP16 GEMM kernels on an RTX 3090.
- What you can do after
- Evaluate GEMM kernels by comparing TFLOPS with compute and DRAM throughput at relevant workload sizes.
Read how CUDA for LLMs explains floating-point operations per second →
Also covered in
- Robotics for ProgrammersIn early access (MEAP)
- Graph Neural Networks in Action
What to know first
- cachingCaching is the temporary storage of data or computed results for reuse, reducing the work or delay of retrieving or computing them again.
- float16float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
Related concepts
- computemeasures · Compute is the processing capacity, resources, or computational work involved in executing software and other computational workloads.
- matrix-matrix multiplicationmeasures · Matrix multiplication is an operation on compatible matrices in which each entry of the product is the dot product of a row of the first matrix and a column of the second.
- floating-point operation countmeasures · A floating-point operation count is the number of arithmetic operations performed on values represented in floating-point format by a computation.
Questions about floating-point operations per second
What is floating-point operations per second?
Floating-point operations per second (FLOP/s) is a measure of computational throughput expressed as the number of floating-point operations performed per second.
Which Manning books explain floating-point operations per second?
Grokking GPU Programming, GPU Programming with Triton and CUDA for LLMs.
What should I know before learning floating-point operations per second?
Manning books that teach floating-point operations per second usually introduce caching and float16 first.
What is floating-point operations per second a kind of?
Floating-point operations per second is a kind of operations per second. Operations per second is a throughput metric measuring the number of completed operations during one second.
What is floating-point operations per second also called?
Floating-point operations per second is also called flops or tflops.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


