Floating-point operation count
A floating-point operation count is the number of arithmetic operations performed on values represented in floating-point format by a computation.
Introduction to Generative AI, Second Edition
FLOPs are floating-point operations used to quantify computational work performed by a system or model.
- How it is used
- Uses training FLOPs to characterize computational scale and set a systemic-risk threshold for general-purpose AI models.
- How deep it goes
- Defines operation counts and performance rates, and discusses regulatory trade-offs without detailing how to count operations.
- Examples
- Compares GPT-3’s estimated training operations with those of more recent state-of-the-art models.
- What you can do after
- Evaluate what a training FLOP count indicates and why it cannot directly establish a model’s capabilities or risks.
Read how Introduction to Generative AI, Second Edition explains floating-point operation count →
Grokking GPU Programming
A floating-point operation is an arithmetic operation performed on floating-point values.
- How it is used
- Uses floating-point operation counts to compare computation with data movement when assessing CUDA kernel performance.
- How deep it goes
- Defines FLOPs, distinguishes them from FLOPS, and works through operation-per-byte calculations for simple and matrix kernels.
- Examples
- Counts an increment as one addition and a matrix thread’s 1,024 multiply-adds as 2,048 operations.
- What you can do after
- Evaluate whether a kernel is memory-bound by relating its floating-point operation count to bytes transferred.
Read how Grokking GPU Programming explains floating-point operation count →
Graph Neural Networks in Action
A floating-point operation is a numerical operation on floating-point values, with FLOPs denoting the count required by a computation.
- How it is used
- Uses FLOPs to compare computational scalability alongside memory, epoch time, and convergence speed.
- How deep it goes
- Defines the metric and demonstrates model profiling, but does not derive operation counts internally.
- Examples
- Uses a PyTorch model, a random input tensor, and THOP's profile function.
- What you can do after
- Apply via THOP's profile function to estimate a model's reported operation count.
Read how Graph Neural Networks in Action explains floating-point operation count →
GPU Programming with Triton
A floating-point operation count is the total number of floating-point operations required by an algorithm or program.
- How it is used
- Quantifies algorithmic work and distinguishes it from processor throughput when comparing GPU performance.
- How deep it goes
- Defines individual operations and derives matrix-multiplication work, without detailing operation counts for arbitrary kernels.
- Examples
- Counts a loop of 1,000 additions and derives work for a matrix product.
- What you can do after
- Evaluate computational work by counting operations and calculate achieved TFLOP/s from runtime.
Read how GPU Programming with Triton explains floating-point operation count →
Also covered in
What to know first
- GPU memoryGPU memory is memory available to a graphics processing unit for storing data and state used in graphics or computation.
- matrixA matrix is a rectangular two-dimensional array of elements arranged in rows and columns.
- floating-point operations per secondFloating-point operations per second (FLOP/s) is a measure of computational throughput expressed as the number of floating-point operations performed per second.
Related concepts
- language modelmeasures · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- arithmetic intensitymeasured by · Arithmetic intensity is the ratio of computational operations performed to bytes of data transferred by a computation.
- pretrainingconfigures · Pretraining is the initial training of a model on broad data to learn general patterns or capabilities before adaptation to specific tasks or domains.
- floating-point arithmeticmeasures · Floating-point arithmetic is numerical computation using finite-precision representations of real numbers, which can introduce rounding and range errors.
Questions about floating-point operation count
What is floating-point operation count?
A floating-point operation count is the number of arithmetic operations performed on values represented in floating-point format by a computation.
Which Manning books explain floating-point operation count?
Introduction to Generative AI, Second Edition, Grokking GPU Programming, Graph Neural Networks in Action and GPU Programming with Triton.
What should I know before learning floating-point operation count?
Manning books that teach floating-point operation count usually introduce GPU memory, matrix and floating-point operations per second first.
What is floating-point operation count a kind of?
Floating-point operation count is a kind of operation count. An operation count is the number of computational operations performed by a model, program, kernel, or algorithm.
What is floating-point operation count also called?
Floating-point operation count is also called flops.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



