liveBook · Manning

Numerical stability

Also called numerically stable

Numerical stability is the property of a computation that limits disproportionate errors or loss of accuracy caused by small perturbations and finite-precision arithmetic.

How 3 Manning books explain it

CUDA for LLMs

Numerical stability is the property of a computation that limits errors or invalid results caused by finite-precision arithmetic.

How it is used
Uses stability to guide numerical choices in CUDA transformer kernels and inference.
How deep it goes
Works through softmax rescaling and accumulation tradeoffs, with guidance for validating kernel error.
Examples
Illustrates stability with an online softmax kernel, a LayerNorm variance calculation, and FP16 inputs accumulated in FP32.
What you can do after
Diagnose numerical risks by comparing custom kernel outputs with CPU references and measuring errors across inputs.

Read how CUDA for LLMs explains numerical stability →

Build a Reasoning Model (From Scratch)

The property of a computation remaining accurate and avoiding underflow or rounding errors when values are very small or very large.

How it is used
Supports response ranking and sequence scoring by replacing tiny probabilities with log probabilities.
How deep it goes
Explains the scaling step, demonstrates torch.log_softmax, and derives joint sequence scores by summing token log probabilities.
Examples
Uses a softmax/log-softmax toy array, plotted logits and probabilities, and Berlin-versus-Bridge next-token scoring.
What you can do after
Apply log scaling to score tokens and compare sequences without losing signal from very small probabilities.

Read how Build a Reasoning Model (From Scratch) explains numerical stability →

GPU Programming with Triton

The property of a numerical computation in which small input or rounding errors do not cause disproportionately large errors in its results.

How it is used
Prevents exponential overflow when computing softmax over logits.
How deep it goes
Explains why softmax uses a maximum reduction alongside its denominator sum, without detailing a complete implementation.
Examples
Uses class-confidence logits and query-key similarities as softmax inputs.
What you can do after
Evaluate why softmax needs a maximum reduction to prevent exponential overflow.

Read how GPU Programming with Triton explains numerical stability →

Also covered in

What to know first

Related concepts

Questions about numerical stability

What is numerical stability?

Numerical stability is the property of a computation that limits disproportionate errors or loss of accuracy caused by small perturbations and finite-precision arithmetic.

Which Manning books explain numerical stability?

CUDA for LLMs, Build a Reasoning Model (From Scratch) and GPU Programming with Triton.

What should I know before learning numerical stability?

Manning books that teach numerical stability usually introduce exponential, mean, linear layer and softmax first.

What is the difference between numerical stability and joint probability distribution?

Numerical stability is the property of a computation that limits disproportionate errors or loss of accuracy caused by small perturbations and finite-precision arithmetic. A joint probability distribution assigns probabilities or probability densities to combinations of values of two or more random variables.

What is numerical stability a kind of?

Numerical stability is a kind of stability. Stability is the property of a system, process, or data pattern of maintaining reliable, bounded, or otherwise consistent behavior or characteristics over time and under relevant changes or perturbations.

What is numerical stability also called?

Numerical stability is also called numerically stable.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.