Bfloat16
bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.
Quantization and Fast Inference
A 16-bit floating-point format with 8 exponent bits and 7 fraction bits, giving it a dynamic range similar to FP32.
- How it is used
- Distinguishes 16-bit training formats when assessing numerical range, memory traffic, and quantization.
- How deep it goes
- Explains the exponent–mantissa tradeoff with range and precision comparisons, without implementing BF16 arithmetic.
- Examples
- Uses a 7-billion-parameter model and Hugging Face checkpoint loading to illustrate BF16 weight storage.
- What you can do after
- Evaluate the range, precision, and storage tradeoffs between BF16 and FP16 for model training and inference.
Read how Quantization and Fast Inference explains bfloat16 →
Build a DeepSeek Model (From Scratch)
BF16 is a 16-bit floating-point numerical format with a wide exponent range commonly used for machine-learning computation.
- How it is used
- Supports memory-efficient machine-learning computation and stable mixed-precision training.
- How deep it goes
- Defines the bit layout, compares numerical trade-offs, and traces BF16 through linear-layer computation and KV-cache sizing.
- Examples
- Uses bit-layout comparisons, FP8 linear-layer flows, and a KV-cache formula involving BF16's two-byte storage.
- What you can do after
- Evaluate memory, range, and precision trade-offs when selecting BF16 for training, inference storage, or cache design.
Read how Build a DeepSeek Model (From Scratch) explains bfloat16 →
Rearchitecting LLMs
bfloat16 is a 16-bit floating-point format with a dynamic range similar to float32 and reduced precision and storage requirements.
- How it is used
- Uses bfloat16 as the selected training precision in QLoRA fine-tuning.
- How deep it goes
- Compares dynamic range and overflow behavior with Float32 and Float16, without detailing format internals.
- Examples
- Shows `bf16=True` and `fp16=False` in an SFTConfig used for training on an L4 GPU.
- What you can do after
- Configure QLoRA training to use bfloat16 on an L4 GPU that supports it natively.
Deep Learning with JAX
A 16-bit floating-point format with a float32-like exponent range and reduced significand precision, commonly used for deep learning.
- How it is used
- Uses bfloat16 for lower-precision deep-learning arrays and inference-oriented storage savings.
- How deep it goes
- Defines the format, demonstrates construction and storage, and tests promotion with float16; it stops short of training details.
- Examples
- Demonstrates a ten-element bfloat16 array, its 20-byte size, and additions with bfloat16 and float16.
- What you can do after
- Operate JAX arrays with bfloat16 and predict result dtypes when combining bfloat16 and float16 operands.
Deep Learning with R, Third Edition
bfloat16 is a 16-bit floating-point data type with an 8-bit exponent and a 7-bit mantissa, providing greater numerical range but lower precision than float16.
- How it is used
- Supports lower-memory, potentially faster inference and computation on hardware optimized for reduced-precision operations.
- How deep it goes
- Explains bit allocation and range-versus-resolution trade-offs, but stops short of detailed floating-point implementation.
- Examples
- Uses a float16 comparison table and recommends testing both formats on the target device.
- What you can do after
- Evaluate bfloat16 against float16 when selecting an inference precision for a newer GPU or TPU.
Read how Deep Learning with R, Third Edition explains bfloat16 →
Domain-Specific Small Language Models
A 16-bit floating-point format with a wider exponent range and reduced mantissa precision than FP16, used in machine learning.
- How it is used
- Uses a reduced-precision format for deep learning while retaining a wider dynamic range than FP16.
- How deep it goes
- Defines the bit allocation and compares range, precision, memory use, and performance with FP16 mixed precision.
- Examples
- Uses a BF16 bit-representation figure and a PyTorch GPU-support check.
- What you can do after
- Apply BF16 through PyTorch by checking whether an NVIDIA GPU supports the format.
Read how Domain-Specific Small Language Models explains bfloat16 →
Also covered in
What to know first
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- throughputThroughput is the rate at which a system processes or completes work, data, or requests over a specified period.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- float16contrasts with · float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
- mixed precisionused by · Mixed precision is a computational technique that uses different numerical precisions for values or operations within a computation.
- QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that combines a quantized pretrained model with trainable low-rank adapters.
- FP32uses · FP32 is a 32-bit single-precision floating-point format for representing approximate real numbers.
- language modelconfigures · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Questions about bfloat16
What is bfloat16?
bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.
Which Manning books explain bfloat16?
Quantization and Fast Inference, Build a DeepSeek Model (From Scratch), Rearchitecting LLMs, Deep Learning with JAX, Deep Learning with R, Third Edition and Domain-Specific Small Language Models.
What should I know before learning bfloat16?
Manning books that teach bfloat16 usually introduce gradient, language model, throughput and transformer architecture first.
What is the difference between bfloat16 and float16?
bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision. float16 is a 16-bit floating-point format, commonly IEEE 754 binary16, with less precision and range than 32-bit floating-point formats.
What are the parts of bfloat16?
Bfloat16 includes exponent bits.
What is bfloat16 also called?
Bfloat16 is also called BF16.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





