liveBook · Manning

Floating-point operation count

Also called flops

A floating-point operation count is the number of arithmetic operations performed on values represented in floating-point format by a computation.

How 4 Manning books explain it

Introduction to Generative AI, Second Edition

FLOPs are floating-point operations used to quantify computational work performed by a system or model.

How it is used
Uses training FLOPs to characterize computational scale and set a systemic-risk threshold for general-purpose AI models.
How deep it goes
Defines operation counts and performance rates, and discusses regulatory trade-offs without detailing how to count operations.
Examples
Compares GPT-3’s estimated training operations with those of more recent state-of-the-art models.
What you can do after
Evaluate what a training FLOP count indicates and why it cannot directly establish a model’s capabilities or risks.

Read how Introduction to Generative AI, Second Edition explains floating-point operation count →

Grokking GPU Programming

A floating-point operation is an arithmetic operation performed on floating-point values.

How it is used
Uses floating-point operation counts to compare computation with data movement when assessing CUDA kernel performance.
How deep it goes
Defines FLOPs, distinguishes them from FLOPS, and works through operation-per-byte calculations for simple and matrix kernels.
Examples
Counts an increment as one addition and a matrix thread’s 1,024 multiply-adds as 2,048 operations.
What you can do after
Evaluate whether a kernel is memory-bound by relating its floating-point operation count to bytes transferred.

Read how Grokking GPU Programming explains floating-point operation count →

Graph Neural Networks in Action

A floating-point operation is a numerical operation on floating-point values, with FLOPs denoting the count required by a computation.

How it is used
Uses FLOPs to compare computational scalability alongside memory, epoch time, and convergence speed.
How deep it goes
Defines the metric and demonstrates model profiling, but does not derive operation counts internally.
Examples
Uses a PyTorch model, a random input tensor, and THOP's profile function.
What you can do after
Apply via THOP's profile function to estimate a model's reported operation count.

Read how Graph Neural Networks in Action explains floating-point operation count →

GPU Programming with Triton

A floating-point operation count is the total number of floating-point operations required by an algorithm or program.

How it is used
Quantifies algorithmic work and distinguishes it from processor throughput when comparing GPU performance.
How deep it goes
Defines individual operations and derives matrix-multiplication work, without detailing operation counts for arbitrary kernels.
Examples
Counts a loop of 1,000 additions and derives work for a matrix product.
What you can do after
Evaluate computational work by counting operations and calculate achieved TFLOP/s from runtime.

Read how GPU Programming with Triton explains floating-point operation count →

Also covered in

What to know first

Related concepts

Questions about floating-point operation count

What is floating-point operation count?

A floating-point operation count is the number of arithmetic operations performed on values represented in floating-point format by a computation.

Which Manning books explain floating-point operation count?

Introduction to Generative AI, Second Edition, Grokking GPU Programming, Graph Neural Networks in Action and GPU Programming with Triton.

What should I know before learning floating-point operation count?

Manning books that teach floating-point operation count usually introduce GPU memory, matrix and floating-point operations per second first.

What is floating-point operation count a kind of?

Floating-point operation count is a kind of operation count. An operation count is the number of computational operations performed by a model, program, kernel, or algorithm.

What is floating-point operation count also called?

Floating-point operation count is also called flops.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.