How deeply does Deep Learning with JAX cover bfloat16?

Defines the format, demonstrates construction and storage, and tests promotion with float16; it stops short of training details.

Bfloat16

As explained in Deep Learning with JAX

A 16-bit floating-point format with a float32-like exponent range and reduced significand precision, commonly used for deep learning.

Need lower-memory floating-point arrays for deep learning? bfloat16 gives JAX a 16-bit format with a float32-like exponent range and reduced significand precision. It uses less storage than float32 and sits alongside float16, float32, and float64 in JAX's dtype system. section 3.3.2

Create and inspect a bfloat16 array

xb16 = jnp.array(range(10), dtype=jnp.bfloat16) xb16.dtype >>> dtype(bfloat16) xb16.nbytes >>> 20 A ten-element array therefore reports the bfloat16 dtype and occupies 20 bytes. Mixing xb16 with float16 promotes the result to float32, while adding two bfloat16 arrays preserves bfloat16. section 3.3.2

How Deep Learning with JAX covers it

liveBook manning.com

Read Deep Learning with JAX on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Deep Learning with R, Third Editionbfloat16 is a 16-bit floating-point data type with an 8-bit exponent and a 7-bit mantissa, providing greater numerical range but lower precision than float16.
liveBook manning.com

Domain-Specific Small Language ModelsA 16-bit floating-point format with a wider exponent range and reduced mantissa precision than FP16, used in machine learning.

Compare how 3 books explain bfloat16 →

Related concepts

float16contrasts with · Float16 is a 16-bit floating-point numeric format that represents values with less precision and storage than single-precision formats such as float32.
mixed precisionused by · Mixed precision is a computational technique that uses multiple numerical precisions for different values or operations to balance resource efficiency and numerical accuracy or stability.
QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
FP32uses · FP32 is the IEEE 754 binary32 single-precision floating-point format, which represents approximate real numbers using 32 bits.
language modelconfigures · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with JAX. The section links point to the parts of the book it draws on.