liveBook · Manning

Bfloat16

Also called BF16

bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.

How 6 Manning books explain it

Quantization and Fast Inference

A 16-bit floating-point format with 8 exponent bits and 7 fraction bits, giving it a dynamic range similar to FP32.

How it is used
Distinguishes 16-bit training formats when assessing numerical range, memory traffic, and quantization.
How deep it goes
Explains the exponent–mantissa tradeoff with range and precision comparisons, without implementing BF16 arithmetic.
Examples
Uses a 7-billion-parameter model and Hugging Face checkpoint loading to illustrate BF16 weight storage.
What you can do after
Evaluate the range, precision, and storage tradeoffs between BF16 and FP16 for model training and inference.

Read how Quantization and Fast Inference explains bfloat16 →

Build a DeepSeek Model (From Scratch)

BF16 is a 16-bit floating-point numerical format with a wide exponent range commonly used for machine-learning computation.

How it is used
Supports memory-efficient machine-learning computation and stable mixed-precision training.
How deep it goes
Defines the bit layout, compares numerical trade-offs, and traces BF16 through linear-layer computation and KV-cache sizing.
Examples
Uses bit-layout comparisons, FP8 linear-layer flows, and a KV-cache formula involving BF16's two-byte storage.
What you can do after
Evaluate memory, range, and precision trade-offs when selecting BF16 for training, inference storage, or cache design.

Read how Build a DeepSeek Model (From Scratch) explains bfloat16 →

Rearchitecting LLMs

bfloat16 is a 16-bit floating-point format with a dynamic range similar to float32 and reduced precision and storage requirements.

How it is used
Uses bfloat16 as the selected training precision in QLoRA fine-tuning.
How deep it goes
Compares dynamic range and overflow behavior with Float32 and Float16, without detailing format internals.
Examples
Shows `bf16=True` and `fp16=False` in an SFTConfig used for training on an L4 GPU.
What you can do after
Configure QLoRA training to use bfloat16 on an L4 GPU that supports it natively.

Read how Rearchitecting LLMs explains bfloat16 →

Deep Learning with JAX

A 16-bit floating-point format with a float32-like exponent range and reduced significand precision, commonly used for deep learning.

How it is used
Uses bfloat16 for lower-precision deep-learning arrays and inference-oriented storage savings.
How deep it goes
Defines the format, demonstrates construction and storage, and tests promotion with float16; it stops short of training details.
Examples
Demonstrates a ten-element bfloat16 array, its 20-byte size, and additions with bfloat16 and float16.
What you can do after
Operate JAX arrays with bfloat16 and predict result dtypes when combining bfloat16 and float16 operands.

Read how Deep Learning with JAX explains bfloat16 →

Deep Learning with R, Third Edition

bfloat16 is a 16-bit floating-point data type with an 8-bit exponent and a 7-bit mantissa, providing greater numerical range but lower precision than float16.

How it is used
Supports lower-memory, potentially faster inference and computation on hardware optimized for reduced-precision operations.
How deep it goes
Explains bit allocation and range-versus-resolution trade-offs, but stops short of detailed floating-point implementation.
Examples
Uses a float16 comparison table and recommends testing both formats on the target device.
What you can do after
Evaluate bfloat16 against float16 when selecting an inference precision for a newer GPU or TPU.

Read how Deep Learning with R, Third Edition explains bfloat16 →

Domain-Specific Small Language Models

A 16-bit floating-point format with a wider exponent range and reduced mantissa precision than FP16, used in machine learning.

How it is used
Uses a reduced-precision format for deep learning while retaining a wider dynamic range than FP16.
How deep it goes
Defines the bit allocation and compares range, precision, memory use, and performance with FP16 mixed precision.
Examples
Uses a BF16 bit-representation figure and a PyTorch GPU-support check.
What you can do after
Apply BF16 through PyTorch by checking whether an NVIDIA GPU supports the format.

Read how Domain-Specific Small Language Models explains bfloat16 →

Also covered in

What to know first

Related concepts

Questions about bfloat16

What is bfloat16?

bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.

Which Manning books explain bfloat16?

Quantization and Fast Inference, Build a DeepSeek Model (From Scratch), Rearchitecting LLMs, Deep Learning with JAX, Deep Learning with R, Third Edition and Domain-Specific Small Language Models.

What should I know before learning bfloat16?

Manning books that teach bfloat16 usually introduce gradient, language model, throughput and transformer architecture first.

What is the difference between bfloat16 and float16?

bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision. float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.

What are the parts of bfloat16?

Bfloat16 includes exponent bits.

What is bfloat16 also called?

Bfloat16 is also called BF16.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.